2026-03-09 18:20:42.880156 | Job console starting... 2026-03-09 18:20:42.903129 | Updating repositories 2026-03-09 18:20:43.020570 | Preparing job workspace 2026-03-09 18:20:55.907289 | Running Ansible setup... 2026-03-09 18:21:05.039298 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master] 2026-03-09 18:21:05.701891 | 2026-03-09 18:21:05.702041 | PLAY [localhost] 2026-03-09 18:21:05.712403 | 2026-03-09 18:21:05.712484 | TASK [Gathering Facts] 2026-03-09 18:21:06.780490 | localhost | ok 2026-03-09 18:21:06.795302 | 2026-03-09 18:21:06.795448 | TASK [log-inventory : Ensure Zuul Ansible directory exists] 2026-03-09 18:21:07.230530 | localhost -> localhost | changed 2026-03-09 18:21:07.243430 | 2026-03-09 18:21:07.243638 | TASK [log-inventory : Copy ansible inventory to logs dir] 2026-03-09 18:21:08.157523 | localhost -> localhost | changed 2026-03-09 18:21:08.166738 | 2026-03-09 18:21:08.166865 | TASK [Setup log path fact] 2026-03-09 18:21:08.184637 | localhost | ok 2026-03-09 18:21:08.197149 | 2026-03-09 18:21:08.197257 | TASK [set-zuul-log-path-fact : Set log path for a build] 2026-03-09 18:21:08.225154 | localhost | ok 2026-03-09 18:21:08.233605 | 2026-03-09 18:21:08.233716 | TASK [emit-job-header : Print job information] 2026-03-09 18:21:08.272547 | # Job Information 2026-03-09 18:21:08.272704 | Ansible Version: 2.15.12 2026-03-09 18:21:08.272854 | Job: watcher-operator-kuttl 2026-03-09 18:21:08.272889 | Pipeline: github-check 2026-03-09 18:21:08.272910 | Executor: ze01.softwarefactory-project.io 2026-03-09 18:21:08.272930 | Triggered by: https://github.com/openstack-k8s-operators/watcher-operator/pull/351 2026-03-09 18:21:08.272949 | Log URL (when completed): https://logserver.rdoproject.org/e72/rdoproject.org/e72ba1c8eae94a9da88ed53f3a62dcb0/ 2026-03-09 18:21:08.272968 | Event ID: 4b6a5c20-1bdd-11f1-9203-7a3e871a787c 2026-03-09 18:21:08.276812 | 2026-03-09 18:21:08.276879 | LOOP [emit-job-header : Print node information] 2026-03-09 18:21:08.390956 | localhost | ok: 2026-03-09 18:21:08.391253 | localhost | # Node Information 2026-03-09 18:21:08.391282 | localhost | Inventory Hostname: controller 2026-03-09 18:21:08.391305 | localhost | Hostname: np0005642801 2026-03-09 18:21:08.391324 | localhost | Username: zuul 2026-03-09 18:21:08.391343 | localhost | Distro: CentOS 9 2026-03-09 18:21:08.391360 | localhost | Provider: vexxhost-nodepool-tripleo 2026-03-09 18:21:08.391377 | localhost | Region: RegionOne 2026-03-09 18:21:08.391393 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost-medium 2026-03-09 18:21:08.391409 | localhost | Product Name: OpenStack Nova 2026-03-09 18:21:08.391425 | localhost | Interface IP: 38.102.83.2 2026-03-09 18:21:08.535804 | localhost | ok: 2026-03-09 18:21:08.535905 | localhost | # Node Information 2026-03-09 18:21:08.535930 | localhost | Inventory Hostname: crc 2026-03-09 18:21:08.535950 | localhost | Hostname: crc 2026-03-09 18:21:08.535968 | localhost | Username: core 2026-03-09 18:21:08.535985 | localhost | Distro: RedHat 4.18 2026-03-09 18:21:08.536002 | localhost | Provider: vexxhost-nodepool-tripleo 2026-03-09 18:21:08.536043 | localhost | Region: RegionOne 2026-03-09 18:21:08.536063 | localhost | Label: crc-cloud-ocp-4-18-1-3xl 2026-03-09 18:21:08.536080 | localhost | Product Name: OpenStack Nova 2026-03-09 18:21:08.536096 | localhost | Interface IP: 38.102.83.74 2026-03-09 18:21:08.590599 | 2026-03-09 18:21:08.590824 | PLAY [all] 2026-03-09 18:21:08.610103 | 2026-03-09 18:21:08.610320 | TASK [Gather network facts] 2026-03-09 18:21:09.035356 | controller | ok 2026-03-09 18:21:09.302007 | crc | ok 2026-03-09 18:21:09.350732 | 2026-03-09 18:21:09.350882 | TASK [include_role : start-zuul-console] 2026-03-09 18:21:09.378728 | controller | ok 2026-03-09 18:21:09.385957 | crc | ok 2026-03-09 18:21:09.405409 | 2026-03-09 18:21:09.405520 | TASK [start-zuul-console : Start zuul_console daemon.] 2026-03-09 18:21:09.880125 | controller | ok 2026-03-09 18:21:09.889826 | crc | ok 2026-03-09 18:21:09.913388 | 2026-03-09 18:21:09.913529 | TASK [include_role : add-build-sshkey] 2026-03-09 18:21:09.937302 | controller | ok 2026-03-09 18:21:09.953413 | crc | ok 2026-03-09 18:21:10.006192 | 2026-03-09 18:21:10.006307 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build] 2026-03-09 18:21:10.267729 | controller -> localhost | ok 2026-03-09 18:21:10.275601 | 2026-03-09 18:21:10.275765 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID] 2026-03-09 18:21:10.302913 | controller | ok 2026-03-09 18:21:10.319896 | crc | ok 2026-03-09 18:21:10.343287 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml 2026-03-09 18:21:10.343387 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml 2026-03-09 18:21:10.350792 | 2026-03-09 18:21:10.350912 | TASK [add-build-sshkey : Create Temp SSH key] 2026-03-09 18:21:11.085668 | controller -> localhost | Generating public/private rsa key pair. 2026-03-09 18:21:11.085865 | controller -> localhost | Your identification has been saved in /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/e72ba1c8eae94a9da88ed53f3a62dcb0_id_rsa. 2026-03-09 18:21:11.085920 | controller -> localhost | Your public key has been saved in /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/e72ba1c8eae94a9da88ed53f3a62dcb0_id_rsa.pub. 2026-03-09 18:21:11.085943 | controller -> localhost | The key fingerprint is: 2026-03-09 18:21:11.085962 | controller -> localhost | SHA256:E8lfm5LOdlAu98PNOtXtbxMutUEwZUpw6f2LpAk+dk0 zuul-build-sshkey 2026-03-09 18:21:11.085981 | controller -> localhost | The key's randomart image is: 2026-03-09 18:21:11.085999 | controller -> localhost | +---[RSA 3072]----+ 2026-03-09 18:21:11.086054 | controller -> localhost | | ..ooo | 2026-03-09 18:21:11.086091 | controller -> localhost | | . . o+o | 2026-03-09 18:21:11.086111 | controller -> localhost | | + +.+ | 2026-03-09 18:21:11.086130 | controller -> localhost | | o = + o | 2026-03-09 18:21:11.086148 | controller -> localhost | | S * = . +| 2026-03-09 18:21:11.086167 | controller -> localhost | | = = E *=| 2026-03-09 18:21:11.086186 | controller -> localhost | | . = B BoB| 2026-03-09 18:21:11.086204 | controller -> localhost | | = = +.Bo| 2026-03-09 18:21:11.086222 | controller -> localhost | | . o oo+| 2026-03-09 18:21:11.086239 | controller -> localhost | +----[SHA256]-----+ 2026-03-09 18:21:11.086312 | controller -> localhost | ok: Runtime: 0:00:00.326092 2026-03-09 18:21:11.092613 | 2026-03-09 18:21:11.092712 | TASK [add-build-sshkey : Remote setup ssh keys (linux)] 2026-03-09 18:21:11.126887 | controller | ok 2026-03-09 18:21:11.136768 | crc | ok 2026-03-09 18:21:11.158732 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml 2026-03-09 18:21:11.158862 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml 2026-03-09 18:21:11.171512 | 2026-03-09 18:21:11.171644 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey] 2026-03-09 18:21:11.199658 | controller | skipping: Conditional result was False 2026-03-09 18:21:11.210654 | crc | skipping: Conditional result was False 2026-03-09 18:21:11.218176 | 2026-03-09 18:21:11.218306 | TASK [add-build-sshkey : Enable access via build key on all nodes] 2026-03-09 18:21:11.643869 | controller | changed 2026-03-09 18:21:11.795821 | crc | changed 2026-03-09 18:21:11.801644 | 2026-03-09 18:21:11.801766 | TASK [add-build-sshkey : Make sure user has a .ssh] 2026-03-09 18:21:12.047492 | controller | ok 2026-03-09 18:21:12.118427 | crc | ok 2026-03-09 18:21:12.124244 | 2026-03-09 18:21:12.124356 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes] 2026-03-09 18:21:12.803612 | controller | changed 2026-03-09 18:21:13.031152 | crc | changed 2026-03-09 18:21:13.036741 | 2026-03-09 18:21:13.036846 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes] 2026-03-09 18:21:13.687428 | controller | changed 2026-03-09 18:21:13.917067 | crc | changed 2026-03-09 18:21:13.925322 | 2026-03-09 18:21:13.925445 | TASK [add-build-sshkey : Remote setup ssh keys (windows)] 2026-03-09 18:21:13.960804 | controller | skipping: Conditional result was False 2026-03-09 18:21:13.971696 | crc | skipping: Conditional result was False 2026-03-09 18:21:13.978071 | 2026-03-09 18:21:13.978170 | TASK [remove-zuul-sshkey : Remove master key from local agent] 2026-03-09 18:21:14.358384 | controller -> localhost | changed 2026-03-09 18:21:14.384606 | 2026-03-09 18:21:14.384739 | TASK [add-build-sshkey : Add back temp key] 2026-03-09 18:21:14.696914 | controller -> localhost | Identity added: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/e72ba1c8eae94a9da88ed53f3a62dcb0_id_rsa (zuul-build-sshkey) 2026-03-09 18:21:14.697134 | controller -> localhost | ok: Runtime: 0:00:00.010529 2026-03-09 18:21:14.703362 | 2026-03-09 18:21:14.703458 | TASK [add-build-sshkey : Verify we can still SSH to all nodes] 2026-03-09 18:21:15.096914 | controller | ok 2026-03-09 18:21:15.141666 | crc | ok 2026-03-09 18:21:15.148420 | 2026-03-09 18:21:15.148511 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)] 2026-03-09 18:21:15.173229 | controller | skipping: Conditional result was False 2026-03-09 18:21:15.194753 | crc | skipping: Conditional result was False 2026-03-09 18:21:15.226198 | 2026-03-09 18:21:15.226348 | TASK [include_role : validate-host] 2026-03-09 18:21:15.251814 | controller | ok 2026-03-09 18:21:15.273536 | crc | ok 2026-03-09 18:21:15.313355 | 2026-03-09 18:21:15.313480 | TASK [validate-host : Define zuul_info_dir fact] 2026-03-09 18:21:15.356103 | controller | ok 2026-03-09 18:21:15.373879 | crc | ok 2026-03-09 18:21:15.379078 | 2026-03-09 18:21:15.379141 | TASK [validate-host : Ensure Zuul Ansible directory exists] 2026-03-09 18:21:15.660940 | controller -> localhost | ok 2026-03-09 18:21:15.668650 | 2026-03-09 18:21:15.668776 | TASK [validate-host : Collect information about the host] 2026-03-09 18:21:16.379842 | controller | ok 2026-03-09 18:21:16.706656 | crc | ok 2026-03-09 18:21:16.719440 | 2026-03-09 18:21:16.719558 | TASK [validate-host : Sanitize hostname] 2026-03-09 18:21:16.775536 | controller | ok 2026-03-09 18:21:16.791359 | crc | ok 2026-03-09 18:21:16.796235 | 2026-03-09 18:21:16.796310 | TASK [validate-host : Write out all ansible variables/facts known for each host] 2026-03-09 18:21:17.287212 | controller -> localhost | changed 2026-03-09 18:21:17.339276 | crc -> localhost | changed 2026-03-09 18:21:17.350735 | 2026-03-09 18:21:17.350858 | TASK [validate-host : Collect information about zuul worker] 2026-03-09 18:21:17.837945 | controller | ok 2026-03-09 18:21:17.865442 | crc | ok 2026-03-09 18:21:17.870750 | 2026-03-09 18:21:17.870814 | TASK [validate-host : Write out all zuul information for each host] 2026-03-09 18:21:18.642604 | crc -> localhost | changed 2026-03-09 18:21:18.674650 | controller -> localhost | changed 2026-03-09 18:21:18.695564 | 2026-03-09 18:21:18.695683 | TASK [include_role : prepare-workspace-openshift] 2026-03-09 18:21:18.711449 | controller | skipping: Conditional result was False 2026-03-09 18:21:18.717699 | 2026-03-09 18:21:18.717796 | TASK [include_role : remove-zuul-sshkey] 2026-03-09 18:21:18.742901 | controller | skipping: Conditional result was False 2026-03-09 18:21:18.749132 | 2026-03-09 18:21:18.749205 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them] 2026-03-09 18:21:19.032079 | controller | ok: "logs" 2026-03-09 18:21:19.033181 | controller | ok: All items complete 2026-03-09 18:21:19.033228 | 2026-03-09 18:21:19.086166 | crc | ok: "logs" 2026-03-09 18:21:19.238303 | controller | ok: "artifacts" 2026-03-09 18:21:19.325998 | crc | ok: "artifacts" 2026-03-09 18:21:19.434721 | controller | ok: "docs" 2026-03-09 18:21:19.605218 | crc | ok: "docs" 2026-03-09 18:21:19.615878 | 2026-03-09 18:21:19.616451 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist] 2026-03-09 18:21:19.937720 | controller | changed: "logs" 2026-03-09 18:21:19.938005 | controller | ok: All items complete 2026-03-09 18:21:19.938078 | 2026-03-09 18:21:19.965450 | crc | changed: "logs" 2026-03-09 18:21:20.117715 | controller | changed: "artifacts" 2026-03-09 18:21:20.207216 | crc | changed: "artifacts" 2026-03-09 18:21:20.334361 | controller | changed: "docs" 2026-03-09 18:21:20.494185 | crc | changed: "docs" 2026-03-09 18:21:20.562595 | 2026-03-09 18:21:20.562712 | PLAY RECAP 2026-03-09 18:21:20.562760 | controller | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-09 18:21:20.562787 | crc | ok: 17 changed: 6 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0 2026-03-09 18:21:20.562810 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-09 18:21:20.562829 | 2026-03-09 18:21:23.210474 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master] 2026-03-09 18:21:23.218868 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/pre.yaml@master] 2026-03-09 18:21:23.945709 | 2026-03-09 18:21:23.945851 | PLAY [all] 2026-03-09 18:21:23.981684 | 2026-03-09 18:21:23.981813 | TASK [mirror-info-fork : Set mirror_fqdn if not defined] 2026-03-09 18:21:24.043515 | controller | ok 2026-03-09 18:21:24.049965 | crc | ok 2026-03-09 18:21:24.054828 | 2026-03-09 18:21:24.054925 | TASK [mirror-info-fork : Create /etc/ci] 2026-03-09 18:21:24.465278 | controller | changed 2026-03-09 18:21:24.585164 | crc | changed 2026-03-09 18:21:24.591134 | 2026-03-09 18:21:24.591205 | TASK [mirror-info-fork : Install ci_mirror script] 2026-03-09 18:21:25.494750 | controller | changed 2026-03-09 18:21:25.633126 | crc | changed 2026-03-09 18:21:25.673924 | 2026-03-09 18:21:25.674060 | LOOP [add-authorized-keys : Enable access via build key on all nodes] 2026-03-09 18:21:26.208221 | controller | changed: 2026-03-09 18:21:26.210363 | controller | { 2026-03-09 18:21:26.210451 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Z/c9osaGGtU6X8fgELwfj/yayRurfcKA0HMFfdpPxev2dbwljysMuzoVp4OZmW1gvGtyYPSNRvnzgsaabPNKNo2ym5NToCP6UM+KSe93aln4BcM/24mXChYAbXJQ5Bqq/pIzsGs/pKetQN+vwvMxLOwTvpcsCJBXaa981RKML6xj9l/UZ7IIq1HSEKMvPLxZMWdu0Ut8DkCd5F4nOw9Wgml2uYpDCj5LLCrQQ9ChdOMz8hz6SighhNlRpPkvPaet3OXxr/ytFMu7j7vv06CaEnuMMiY2aTWN1Imin9eHAylIqFHta/3gFfQSWt9jXM7owkBLKL7ATzhaAn+fjNupw== arxcruz@redhat.com" 2026-03-09 18:21:26.210477 | controller | } 2026-03-09 18:21:26.343769 | crc | changed: 2026-03-09 18:21:26.343877 | crc | { 2026-03-09 18:21:26.343901 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Z/c9osaGGtU6X8fgELwfj/yayRurfcKA0HMFfdpPxev2dbwljysMuzoVp4OZmW1gvGtyYPSNRvnzgsaabPNKNo2ym5NToCP6UM+KSe93aln4BcM/24mXChYAbXJQ5Bqq/pIzsGs/pKetQN+vwvMxLOwTvpcsCJBXaa981RKML6xj9l/UZ7IIq1HSEKMvPLxZMWdu0Ut8DkCd5F4nOw9Wgml2uYpDCj5LLCrQQ9ChdOMz8hz6SighhNlRpPkvPaet3OXxr/ytFMu7j7vv06CaEnuMMiY2aTWN1Imin9eHAylIqFHta/3gFfQSWt9jXM7owkBLKL7ATzhaAn+fjNupw== arxcruz@redhat.com" 2026-03-09 18:21:26.343921 | crc | } 2026-03-09 18:21:26.469778 | controller | changed: 2026-03-09 18:21:26.469932 | controller | { 2026-03-09 18:21:26.469958 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS4Fn6k4deCnIlOtLWqZJyksbepjQt04j8Ed8CGx9EKkj0fKiAxiI4TadXQYPuNHMixZy4Nevjb6aDhL5Z906TfvNHKUrjrG7G26a0k8vdc61NEQ7FmcGMWRLwwc6ReDO7lFpzYKBMk4YqfWgBuGU/K6WLKiVW2cVvwIuGIaYrE1OiiX0iVUUk7KApXlDJMXn7qjSYynfO4mF629NIp8FJal38+Kv+HA+0QkE5Y2xXnzD4Lar5+keymiCHRntPppXHeLIRzbt0gxC7v3L72hpQ3BTBEzwHpeS8KY+SX1y5lRMN45thCHfJqGmARJREDjBvWG8JXOPmVIKQtZmVcD5b mandreou@redhat.com" 2026-03-09 18:21:26.469978 | controller | } 2026-03-09 18:21:26.681382 | controller | changed: 2026-03-09 18:21:26.681496 | controller | { 2026-03-09 18:21:26.681532 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MiLfy30deHA7xPOAlew5qUq3UP2gmRMYJi8PtkjFB20/DKeWwWNnkZPqP9AayruRoo51SIiVg870gbZE2jYl+Ncx/FYDe56JeC3ySZsXoAVkC9bP7gkOGqOmJjirvAgPMI7bogVz8i+66Q4Ar7OKTp3762G4IuWPPEg4ce4Y7lx9qWocZapHYq4cYKMxrOZ7SEbFSATBbe2bPZAPKTw8do/Eny+Hq/LkHFhIeyra6cqTFQYShr+zPln0Cr+ro/pDX3bB+1ubFgTpjpkkkQsLhDfR6cCdCWM2lgnS3BTtYj5Ct9/JRPR5YOphqZz+uB+OEu2IL68hmU9vNTth1KeX rlandy@redhat.com" 2026-03-09 18:21:26.681575 | controller | } 2026-03-09 18:21:26.696801 | crc | changed: 2026-03-09 18:21:26.696925 | crc | { 2026-03-09 18:21:26.696968 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS4Fn6k4deCnIlOtLWqZJyksbepjQt04j8Ed8CGx9EKkj0fKiAxiI4TadXQYPuNHMixZy4Nevjb6aDhL5Z906TfvNHKUrjrG7G26a0k8vdc61NEQ7FmcGMWRLwwc6ReDO7lFpzYKBMk4YqfWgBuGU/K6WLKiVW2cVvwIuGIaYrE1OiiX0iVUUk7KApXlDJMXn7qjSYynfO4mF629NIp8FJal38+Kv+HA+0QkE5Y2xXnzD4Lar5+keymiCHRntPppXHeLIRzbt0gxC7v3L72hpQ3BTBEzwHpeS8KY+SX1y5lRMN45thCHfJqGmARJREDjBvWG8JXOPmVIKQtZmVcD5b mandreou@redhat.com" 2026-03-09 18:21:26.697005 | crc | } 2026-03-09 18:21:26.894414 | controller | changed: 2026-03-09 18:21:26.894508 | controller | { 2026-03-09 18:21:26.894535 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCbgz8gdERiJlk2IKOtkjQxEXejrio6ZYMJAVJYpOIp raukadah@gmail.com" 2026-03-09 18:21:26.894554 | controller | } 2026-03-09 18:21:27.080376 | crc | changed: 2026-03-09 18:21:27.084719 | crc | { 2026-03-09 18:21:27.084810 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MiLfy30deHA7xPOAlew5qUq3UP2gmRMYJi8PtkjFB20/DKeWwWNnkZPqP9AayruRoo51SIiVg870gbZE2jYl+Ncx/FYDe56JeC3ySZsXoAVkC9bP7gkOGqOmJjirvAgPMI7bogVz8i+66Q4Ar7OKTp3762G4IuWPPEg4ce4Y7lx9qWocZapHYq4cYKMxrOZ7SEbFSATBbe2bPZAPKTw8do/Eny+Hq/LkHFhIeyra6cqTFQYShr+zPln0Cr+ro/pDX3bB+1ubFgTpjpkkkQsLhDfR6cCdCWM2lgnS3BTtYj5Ct9/JRPR5YOphqZz+uB+OEu2IL68hmU9vNTth1KeX rlandy@redhat.com" 2026-03-09 18:21:27.084835 | crc | } 2026-03-09 18:21:27.100726 | controller | changed: 2026-03-09 18:21:27.100827 | controller | { 2026-03-09 18:21:27.100850 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqb3Q/9uDf4LmihQ7xeJ9gA/STIQUFPSfyyV0m8AoQi bshewale@redhat.com" 2026-03-09 18:21:27.100869 | controller | } 2026-03-09 18:21:27.346809 | controller | changed: 2026-03-09 18:21:27.346899 | controller | { 2026-03-09 18:21:27.346923 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8QqQx0Az2ysJt2JuffucLijhBqnsXKEIx5GyHwxVULROa8VtNFXUDH6ZKZavhiMcmfHB2+TBTda+lDP4FldYj06dGmzCY+IYGa+uDRdxHNGYjvCfLFcmLlzRK6fNbTcui+KlUFUdKe0fb9CRoGKyhlJD5GRkM1Dv+Yb6Bj+RNnmm1fVGYxzmrD2utvffYEb0SZGWxq2R9gefx1q/3wCGjeqvufEV+AskPhVGc5T7t9eyZ4qmslkLh1/nMuaIBFcr9AUACRajsvk6mXrAN1g3HlBf2gQlhi1UEyfbqIQvzzFtsbLDlSum/KmKjy818GzvWjERfQ0VkGzCd9bSLVL dviroel@redhat.com" 2026-03-09 18:21:27.346943 | controller | } 2026-03-09 18:21:27.453631 | crc | changed: 2026-03-09 18:21:27.453745 | crc | { 2026-03-09 18:21:27.453769 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCbgz8gdERiJlk2IKOtkjQxEXejrio6ZYMJAVJYpOIp raukadah@gmail.com" 2026-03-09 18:21:27.453787 | crc | } 2026-03-09 18:21:27.602402 | controller | changed: 2026-03-09 18:21:27.602541 | controller | { 2026-03-09 18:21:27.602572 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLOQd4ZLtkZXQGY6UwAr/06ppWQK4fDO3HaqxPk98csyOCBXsliSKK39Bso828+5srIXiW7aI6aC9P5mwi4mUZlGPfJlQbfrcGvY+b/SocuvaGK+1RrHLoJCT52LBhwgrzlXio2jeksZeein8iaTrhsPrOAs7KggIL/rB9hEiB3NaOPWhhoCP4vlW6MEMExGcqB/1FVxXFBPnLkEyW0Lk7ycVflZl2ocRxbfjZi0+tI1Wlinp8PvSQSc/WVrAcDgKjc/mB4ODPOyYy3G8FHgfMsrXSDEyjBKgLKMsdCrAUcqJQWjkqXleXSYOV4q3pzL+9umK+q/e3P/bIoSFQzmJKTU1eDfuvPXmow9F5H54fii/Da7ezlMJ+wPGHJrRAkmzvMbALy7xwswLhZMkOGNtRcPqaKYRmIBKpw3o6bCTtcNUHOtOQnzwY8JzrM2eBWJBXAANYw+9/ho80JIiwhg29CFNpVBuHbql2YxJQNrnl90guN65rYNpDxdIluweyUf8= anbanerj@kaermorhen" 2026-03-09 18:21:27.602593 | controller | } 2026-03-09 18:21:27.848737 | crc | changed: 2026-03-09 18:21:27.850550 | crc | { 2026-03-09 18:21:27.850606 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqb3Q/9uDf4LmihQ7xeJ9gA/STIQUFPSfyyV0m8AoQi bshewale@redhat.com" 2026-03-09 18:21:27.850638 | crc | } 2026-03-09 18:21:27.865972 | controller | changed: 2026-03-09 18:21:27.866076 | controller | { 2026-03-09 18:21:27.866101 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3VwV8Im9kRm49lt3tM36hj4Zv27FxGo4C1Q/0jqhzFmHY7RHbmeRr8ObhwWoHjXSozKWg8FL5ER0z3hTwL0W6lez3sL7hUaCmSuZmG5Hnl3x4vTSxDI9JZ/Y65rtYiiWQo2fC5xJhU/4+0e5e/pseCm8cKRSu+SaxhO+sd6FDojA2x1BzOzKiQRDy/1zWGp/cZkxcEuB1wHI5LMzN03c67vmbu+fhZRAUO4dQkvcnj2LrhQtpa+ytvnSjr8icMDosf1OsbSffwZFyHB/hfWGAfe0eIeSA2XPraxiPknXxiPKx2MJsaUTYbsZcm3EjFdHBBMumw5rBI74zLrMRvCO9GwBEmGT4rFng1nP+yw5DB8sn2zqpOsPg1LYRwCPOUveC13P6pgsZZPh812e8v5EKnETct+5XI3dVpdw6CnNiLwAyVAF15DJvBGT/u1k0Myg/bQn+Gv9k2MSj6LvQmf6WbZu2Wgjm30z3FyCneBqTL7mLF19YXzeC0ufHz5pnO1E= dasm@fedora" 2026-03-09 18:21:27.866122 | controller | } 2026-03-09 18:21:28.116891 | controller | changed: 2026-03-09 18:21:28.117117 | controller | { 2026-03-09 18:21:28.117185 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk cjeanner" 2026-03-09 18:21:28.117236 | controller | } 2026-03-09 18:21:28.218521 | crc | changed: 2026-03-09 18:21:28.218648 | crc | { 2026-03-09 18:21:28.218683 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8QqQx0Az2ysJt2JuffucLijhBqnsXKEIx5GyHwxVULROa8VtNFXUDH6ZKZavhiMcmfHB2+TBTda+lDP4FldYj06dGmzCY+IYGa+uDRdxHNGYjvCfLFcmLlzRK6fNbTcui+KlUFUdKe0fb9CRoGKyhlJD5GRkM1Dv+Yb6Bj+RNnmm1fVGYxzmrD2utvffYEb0SZGWxq2R9gefx1q/3wCGjeqvufEV+AskPhVGc5T7t9eyZ4qmslkLh1/nMuaIBFcr9AUACRajsvk6mXrAN1g3HlBf2gQlhi1UEyfbqIQvzzFtsbLDlSum/KmKjy818GzvWjERfQ0VkGzCd9bSLVL dviroel@redhat.com" 2026-03-09 18:21:28.218711 | crc | } 2026-03-09 18:21:28.348372 | controller | changed: 2026-03-09 18:21:28.348497 | controller | { 2026-03-09 18:21:28.348533 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDercCMGn8rW1C4P67tHgtflPdTeXlpyUJYH+6XDd2lR jgilaber@redhat.com" 2026-03-09 18:21:28.348563 | controller | } 2026-03-09 18:21:28.597992 | crc | changed: 2026-03-09 18:21:28.598191 | crc | { 2026-03-09 18:21:28.598227 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLOQd4ZLtkZXQGY6UwAr/06ppWQK4fDO3HaqxPk98csyOCBXsliSKK39Bso828+5srIXiW7aI6aC9P5mwi4mUZlGPfJlQbfrcGvY+b/SocuvaGK+1RrHLoJCT52LBhwgrzlXio2jeksZeein8iaTrhsPrOAs7KggIL/rB9hEiB3NaOPWhhoCP4vlW6MEMExGcqB/1FVxXFBPnLkEyW0Lk7ycVflZl2ocRxbfjZi0+tI1Wlinp8PvSQSc/WVrAcDgKjc/mB4ODPOyYy3G8FHgfMsrXSDEyjBKgLKMsdCrAUcqJQWjkqXleXSYOV4q3pzL+9umK+q/e3P/bIoSFQzmJKTU1eDfuvPXmow9F5H54fii/Da7ezlMJ+wPGHJrRAkmzvMbALy7xwswLhZMkOGNtRcPqaKYRmIBKpw3o6bCTtcNUHOtOQnzwY8JzrM2eBWJBXAANYw+9/ho80JIiwhg29CFNpVBuHbql2YxJQNrnl90guN65rYNpDxdIluweyUf8= anbanerj@kaermorhen" 2026-03-09 18:21:28.598257 | crc | } 2026-03-09 18:21:28.620118 | controller | changed: 2026-03-09 18:21:28.620227 | controller | { 2026-03-09 18:21:28.620261 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMI6kkg9Wg0sG7jIJmyZemEBwUn1yzNpQQd3gnulOmZ adrianfuscoarnejo@gmail.com" 2026-03-09 18:21:28.620288 | controller | } 2026-03-09 18:21:28.862950 | controller | changed: 2026-03-09 18:21:28.863072 | controller | { 2026-03-09 18:21:28.863098 | controller | "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPijwpQu/3jhhhBZInXNOLEH57DrknPc3PLbsRvYyJIFzwYjX+WD4a7+nGnMYS42MuZk6TJcVqgnqofVx4isoD4= ramishra@redhat.com" 2026-03-09 18:21:28.863117 | controller | } 2026-03-09 18:21:28.969471 | crc | changed: 2026-03-09 18:21:28.969578 | crc | { 2026-03-09 18:21:28.969602 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3VwV8Im9kRm49lt3tM36hj4Zv27FxGo4C1Q/0jqhzFmHY7RHbmeRr8ObhwWoHjXSozKWg8FL5ER0z3hTwL0W6lez3sL7hUaCmSuZmG5Hnl3x4vTSxDI9JZ/Y65rtYiiWQo2fC5xJhU/4+0e5e/pseCm8cKRSu+SaxhO+sd6FDojA2x1BzOzKiQRDy/1zWGp/cZkxcEuB1wHI5LMzN03c67vmbu+fhZRAUO4dQkvcnj2LrhQtpa+ytvnSjr8icMDosf1OsbSffwZFyHB/hfWGAfe0eIeSA2XPraxiPknXxiPKx2MJsaUTYbsZcm3EjFdHBBMumw5rBI74zLrMRvCO9GwBEmGT4rFng1nP+yw5DB8sn2zqpOsPg1LYRwCPOUveC13P6pgsZZPh812e8v5EKnETct+5XI3dVpdw6CnNiLwAyVAF15DJvBGT/u1k0Myg/bQn+Gv9k2MSj6LvQmf6WbZu2Wgjm30z3FyCneBqTL7mLF19YXzeC0ufHz5pnO1E= dasm@fedora" 2026-03-09 18:21:28.969633 | crc | } 2026-03-09 18:21:29.070871 | controller | changed: 2026-03-09 18:21:29.070970 | controller | { 2026-03-09 18:21:29.070993 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWBreHW95Wz2Toz5YwCGQwFcUG8oFYkienDh9tntmDc ralfieri@redhat.com" 2026-03-09 18:21:29.071011 | controller | } 2026-03-09 18:21:29.323962 | controller | changed: 2026-03-09 18:21:29.324119 | controller | { 2026-03-09 18:21:29.324175 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK0iKdi8jQTpQrDdLVH/AAgLVYyTXF7AQ1gjc/5uT3t ykarel@yatinkarel" 2026-03-09 18:21:29.324204 | controller | } 2026-03-09 18:21:29.346187 | crc | changed: 2026-03-09 18:21:29.346289 | crc | { 2026-03-09 18:21:29.346321 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk cjeanner" 2026-03-09 18:21:29.346347 | crc | } 2026-03-09 18:21:29.525559 | controller | changed: 2026-03-09 18:21:29.525673 | controller | { 2026-03-09 18:21:29.525711 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/V/cLotA6LZeO32VL45Hd78skuA2lJA425Sm2LlQeZ fmount@horcrux" 2026-03-09 18:21:29.525744 | controller | } 2026-03-09 18:21:29.732875 | controller | changed: 2026-03-09 18:21:29.732997 | controller | { 2026-03-09 18:21:29.733071 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa7QCjuDMVmRPo1rREbGwzYeBCYVN+Ou/3WKXZEC6Sr" 2026-03-09 18:21:29.733103 | controller | } 2026-03-09 18:21:29.791069 | crc | changed: 2026-03-09 18:21:29.791169 | crc | { 2026-03-09 18:21:29.791193 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDercCMGn8rW1C4P67tHgtflPdTeXlpyUJYH+6XDd2lR jgilaber@redhat.com" 2026-03-09 18:21:29.791212 | crc | } 2026-03-09 18:21:29.951921 | controller | changed: 2026-03-09 18:21:29.952079 | controller | { 2026-03-09 18:21:29.952120 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfNtF7NvKl915TGsGGoseUb06Hj8L/S4toWf0hExeY+F00woL6NvBlJD0nDct+P5a22I4EhvoQCRQ8reaPCm1lybR3uiRIJsj+8zkVvLwby9LXzfZorlNG9ofjd00FEmB09uW/YvTl6Q9XwwwX6tInzIOv3TMqTHHGOL74ibbj8J/FJR0cFEyj0z4WQRvtkh32xAHl83gbuINryMt0sqRI+clj2381NKL55DRLQrVw0gsfqqxiHAnXg21qWmc4J+b9e9kiuAFQjcjwTVkwJCcg3xbPwC/qokYRby/Y5S40UUd7/jEARGXT7RZgpzTuDd1oZiCVrnrqJNPaMNdVv5MLeFdf1B7iIe5aa/fGouX7AO4SdKhZUdnJmCFAGvjC6S3JMZ2wAcUl+OHnssfmdj7XL50cLo27vjuzMtLAgSqi6N99m92WCF2s8J9aVzszX7Xz9OKZCeGsiVJp3/NdABKzSEAyM9xBD/5Vho894Sav+otpySHe3p6RUTgbB5Zu8VyZRZ/UtB3ueXxyo764yrc6qWIDqrehm84Xm9g+/jpIBzGPl07NUNJpdt/6Sgf9RIKXw/7XypO5yZfUcuFNGTxLfqjTNrtgLZNcjfav6sSdVXVcMPL//XNuRdKmVFaO76eV/oGMQGr1fGcCD+N+CpI7+Q+fCNB6VFWG4nZFuI/Iuw== averdagu@redhat.com" 2026-03-09 18:21:29.952151 | controller | } 2026-03-09 18:21:30.133785 | crc | changed: 2026-03-09 18:21:30.133922 | crc | { 2026-03-09 18:21:30.133956 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMI6kkg9Wg0sG7jIJmyZemEBwUn1yzNpQQd3gnulOmZ adrianfuscoarnejo@gmail.com" 2026-03-09 18:21:30.133986 | crc | } 2026-03-09 18:21:30.195936 | controller | changed: 2026-03-09 18:21:30.196115 | controller | { 2026-03-09 18:21:30.196164 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq8l27xI+QlQVdS4djp9ogSoyrNE2+Ox6vKPdhSNL1J3PE5w+WCSvMz9A5gnNuH810zwbekEApbxTze/gLQJwBHA52CChfURpXrFaxY7ePXRElwKAL3mJfzBWY/c5jnNL9TCVmFJTGZkFZP3Nh+BMgZvL6xBkt3WKm6Uq18qzd9XeKcZusrA+O+uLv1fVeQnadY9RIqOCyeFYCzLWrUfTyE8x/XG0hAWIM7qpnF2cALQS2h9n4hW5ybiUN790H08wf9hFwEf5nxY9Z9dVkPFQiTSGKNBzmnCXU9skxS/xhpFjJ5duGSZdtAHe9O+nGZm9c67hxgtf8e5PDuqAdXEv2cf6e3VBAt+Bz8EKI3yosTj0oZHfwr42Yzb1l/SKy14Rggsrc9KAQlrGXan6+u2jcQqqx7l+SWmnpFiWTV9u5cWj2IgOhApOitmRBPYqk9rE2usfO0hLn/Pj/R/Nau4803e1/EikdLE7Ps95s9mX5jRDjAoUa2JwFF5RsVFyL910= ashigupt@ashigupt.remote.csb" 2026-03-09 18:21:30.196211 | controller | } 2026-03-09 18:21:30.401566 | controller | changed: 2026-03-09 18:21:30.401708 | controller | { 2026-03-09 18:21:30.401740 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKLl0NYKwoZ/JY5KeZU8VwRAggeOxqQJeoqp3dsAaY9" 2026-03-09 18:21:30.401766 | controller | } 2026-03-09 18:21:30.452966 | crc | changed: 2026-03-09 18:21:30.453090 | crc | { 2026-03-09 18:21:30.453124 | crc | "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPijwpQu/3jhhhBZInXNOLEH57DrknPc3PLbsRvYyJIFzwYjX+WD4a7+nGnMYS42MuZk6TJcVqgnqofVx4isoD4= ramishra@redhat.com" 2026-03-09 18:21:30.453146 | crc | } 2026-03-09 18:21:30.613464 | controller | changed: 2026-03-09 18:21:30.613566 | controller | { 2026-03-09 18:21:30.613600 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASASQOH2BcOyLKuuDOdWZlPi2orcjcA8q4400T73DLH evallesp@fedora" 2026-03-09 18:21:30.613634 | controller | } 2026-03-09 18:21:30.795131 | crc | changed: 2026-03-09 18:21:30.795226 | crc | { 2026-03-09 18:21:30.795249 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWBreHW95Wz2Toz5YwCGQwFcUG8oFYkienDh9tntmDc ralfieri@redhat.com" 2026-03-09 18:21:30.795268 | crc | } 2026-03-09 18:21:30.833389 | controller | changed: 2026-03-09 18:21:30.833486 | controller | { 2026-03-09 18:21:30.833508 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeBWlamUph+jRKV2qrx1PGU7vWuGIt5+z9k96I8WehW amsinha@amsinha-mac" 2026-03-09 18:21:30.833527 | controller | } 2026-03-09 18:21:31.041770 | controller | changed: 2026-03-09 18:21:31.041875 | controller | { 2026-03-09 18:21:31.041900 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANvVgvJBlK3gb1yz5uef/JqIGq4HLEmY2dYA8e37swb morenod@redhat-laptop" 2026-03-09 18:21:31.041919 | controller | } 2026-03-09 18:21:31.193745 | crc | changed: 2026-03-09 18:21:31.193844 | crc | { 2026-03-09 18:21:31.193867 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK0iKdi8jQTpQrDdLVH/AAgLVYyTXF7AQ1gjc/5uT3t ykarel@yatinkarel" 2026-03-09 18:21:31.193926 | crc | } 2026-03-09 18:21:31.251102 | controller | changed: 2026-03-09 18:21:31.251199 | controller | { 2026-03-09 18:21:31.251222 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZdI7t1cxYx65heVI24HTV4F7oQLW1zyfxHreL2TIJKxjyrUUKIFEUmTutcBlJRLNT2Eoix6x1sOw9YrchloCLcn//SGfTElr9mSc5jbjb7QXEU+zJMhtxyEJ1Po3CUGnj7ckiIXw7wcawZtrEOAQ9pH3ExYCJcEMiyNjRQZCxT3tPK+S4B95EWh5Fsrz9CkwpjNRPPH7LigCeQTM3Wc7r97utAslBUUvYceDSLA7rMgkitJE38b7rZBeYzsGQ8YYUBjTCtehqQXxCRjizbHWaaZkBU+N3zkKB6n/iCNGIO690NK7A/qb6msTijiz1PeuM8ThOsi9qXnbX5v0PoTpcFSojV7NHAQ71f0XXuS43FhZctT+Dcx44dT8Fb5vJu2cJGrk+qF8ZgJYNpRS7gPg0EG2EqjK7JMf9ULdjSu0r+KlqIAyLvtzT4eOnQipoKlb/WG5D/0ohKv7OMQ352ggfkBFIQsRXyyTCT98Ft9juqPuahi3CAQmP4H9dyE+7+Kz437PEtsxLmfm6naNmWi7Ee1DqWPwS8rEajsm4sNM4wW9gdBboJQtc0uZw0DfLj1I9r3Mc8Ol0jYtz0yNQDSzVLrGCaJlC311trU70tZ+ZkAVV6Mn8lOhSbj1cK0lvSr6ZK4dgqGl3I1eTZJJhbLNdg7UOVaiRx9543+C/p/As7w== brjackma@redhat.com" 2026-03-09 18:21:31.251241 | controller | } 2026-03-09 18:21:31.466294 | controller | changed: 2026-03-09 18:21:31.466422 | controller | { 2026-03-09 18:21:31.466464 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwedoZ0TWPJX/z/4TAbO/kKcDZOQVgRH0hAqrL5UCI1 vcastell@redhat.com" 2026-03-09 18:21:31.466499 | controller | } 2026-03-09 18:21:31.553811 | crc | changed: 2026-03-09 18:21:31.553992 | crc | { 2026-03-09 18:21:31.554152 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/V/cLotA6LZeO32VL45Hd78skuA2lJA425Sm2LlQeZ fmount@horcrux" 2026-03-09 18:21:31.554220 | crc | } 2026-03-09 18:21:31.744175 | controller | changed: 2026-03-09 18:21:31.744290 | controller | { 2026-03-09 18:21:31.744326 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmv8sE8GCk6ZTPIqF0FQrttBdL3mq7rCm/IJy0xDFh7 michburk@redhat.com" 2026-03-09 18:21:31.744357 | controller | } 2026-03-09 18:21:32.028308 | controller | changed: 2026-03-09 18:21:32.028944 | controller | { 2026-03-09 18:21:32.029084 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICy6GpGEtwevXEEn4mmLR5lmSLe23dGgAvzkB9DMNbkf rsafrono@rsafrono" 2026-03-09 18:21:32.029119 | controller | } 2026-03-09 18:21:32.040245 | crc | changed: 2026-03-09 18:21:32.040396 | crc | { 2026-03-09 18:21:32.040442 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa7QCjuDMVmRPo1rREbGwzYeBCYVN+Ou/3WKXZEC6Sr" 2026-03-09 18:21:32.040482 | crc | } 2026-03-09 18:21:35.606275 | crc | changed: 2026-03-09 18:21:35.606389 | crc | { 2026-03-09 18:21:35.606415 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfNtF7NvKl915TGsGGoseUb06Hj8L/S4toWf0hExeY+F00woL6NvBlJD0nDct+P5a22I4EhvoQCRQ8reaPCm1lybR3uiRIJsj+8zkVvLwby9LXzfZorlNG9ofjd00FEmB09uW/YvTl6Q9XwwwX6tInzIOv3TMqTHHGOL74ibbj8J/FJR0cFEyj0z4WQRvtkh32xAHl83gbuINryMt0sqRI+clj2381NKL55DRLQrVw0gsfqqxiHAnXg21qWmc4J+b9e9kiuAFQjcjwTVkwJCcg3xbPwC/qokYRby/Y5S40UUd7/jEARGXT7RZgpzTuDd1oZiCVrnrqJNPaMNdVv5MLeFdf1B7iIe5aa/fGouX7AO4SdKhZUdnJmCFAGvjC6S3JMZ2wAcUl+OHnssfmdj7XL50cLo27vjuzMtLAgSqi6N99m92WCF2s8J9aVzszX7Xz9OKZCeGsiVJp3/NdABKzSEAyM9xBD/5Vho894Sav+otpySHe3p6RUTgbB5Zu8VyZRZ/UtB3ueXxyo764yrc6qWIDqrehm84Xm9g+/jpIBzGPl07NUNJpdt/6Sgf9RIKXw/7XypO5yZfUcuFNGTxLfqjTNrtgLZNcjfav6sSdVXVcMPL//XNuRdKmVFaO76eV/oGMQGr1fGcCD+N+CpI7+Q+fCNB6VFWG4nZFuI/Iuw== averdagu@redhat.com" 2026-03-09 18:21:35.606437 | crc | } 2026-03-09 18:21:35.606473 | crc | changed: All items complete 2026-03-09 18:21:35.606495 | 2026-03-09 18:21:36.318670 | crc | changed: 2026-03-09 18:21:36.318791 | crc | { 2026-03-09 18:21:36.318824 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq8l27xI+QlQVdS4djp9ogSoyrNE2+Ox6vKPdhSNL1J3PE5w+WCSvMz9A5gnNuH810zwbekEApbxTze/gLQJwBHA52CChfURpXrFaxY7ePXRElwKAL3mJfzBWY/c5jnNL9TCVmFJTGZkFZP3Nh+BMgZvL6xBkt3WKm6Uq18qzd9XeKcZusrA+O+uLv1fVeQnadY9RIqOCyeFYCzLWrUfTyE8x/XG0hAWIM7qpnF2cALQS2h9n4hW5ybiUN790H08wf9hFwEf5nxY9Z9dVkPFQiTSGKNBzmnCXU9skxS/xhpFjJ5duGSZdtAHe9O+nGZm9c67hxgtf8e5PDuqAdXEv2cf6e3VBAt+Bz8EKI3yosTj0oZHfwr42Yzb1l/SKy14Rggsrc9KAQlrGXan6+u2jcQqqx7l+SWmnpFiWTV9u5cWj2IgOhApOitmRBPYqk9rE2usfO0hLn/Pj/R/Nau4803e1/EikdLE7Ps95s9mX5jRDjAoUa2JwFF5RsVFyL910= ashigupt@ashigupt.remote.csb" 2026-03-09 18:21:36.318856 | crc | } 2026-03-09 18:21:36.694885 | crc | changed: 2026-03-09 18:21:36.694981 | crc | { 2026-03-09 18:21:36.695005 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKLl0NYKwoZ/JY5KeZU8VwRAggeOxqQJeoqp3dsAaY9" 2026-03-09 18:21:36.695085 | crc | } 2026-03-09 18:21:37.109289 | crc | changed: 2026-03-09 18:21:37.109393 | crc | { 2026-03-09 18:21:37.109418 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASASQOH2BcOyLKuuDOdWZlPi2orcjcA8q4400T73DLH evallesp@fedora" 2026-03-09 18:21:37.109438 | crc | } 2026-03-09 18:21:37.501376 | crc | changed: 2026-03-09 18:21:37.501689 | crc | { 2026-03-09 18:21:37.501731 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeBWlamUph+jRKV2qrx1PGU7vWuGIt5+z9k96I8WehW amsinha@amsinha-mac" 2026-03-09 18:21:37.501758 | crc | } 2026-03-09 18:21:37.876763 | crc | changed: 2026-03-09 18:21:37.876896 | crc | { 2026-03-09 18:21:37.876934 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANvVgvJBlK3gb1yz5uef/JqIGq4HLEmY2dYA8e37swb morenod@redhat-laptop" 2026-03-09 18:21:37.876964 | crc | } 2026-03-09 18:21:38.263579 | crc | changed: 2026-03-09 18:21:38.263676 | crc | { 2026-03-09 18:21:38.263707 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZdI7t1cxYx65heVI24HTV4F7oQLW1zyfxHreL2TIJKxjyrUUKIFEUmTutcBlJRLNT2Eoix6x1sOw9YrchloCLcn//SGfTElr9mSc5jbjb7QXEU+zJMhtxyEJ1Po3CUGnj7ckiIXw7wcawZtrEOAQ9pH3ExYCJcEMiyNjRQZCxT3tPK+S4B95EWh5Fsrz9CkwpjNRPPH7LigCeQTM3Wc7r97utAslBUUvYceDSLA7rMgkitJE38b7rZBeYzsGQ8YYUBjTCtehqQXxCRjizbHWaaZkBU+N3zkKB6n/iCNGIO690NK7A/qb6msTijiz1PeuM8ThOsi9qXnbX5v0PoTpcFSojV7NHAQ71f0XXuS43FhZctT+Dcx44dT8Fb5vJu2cJGrk+qF8ZgJYNpRS7gPg0EG2EqjK7JMf9ULdjSu0r+KlqIAyLvtzT4eOnQipoKlb/WG5D/0ohKv7OMQ352ggfkBFIQsRXyyTCT98Ft9juqPuahi3CAQmP4H9dyE+7+Kz437PEtsxLmfm6naNmWi7Ee1DqWPwS8rEajsm4sNM4wW9gdBboJQtc0uZw0DfLj1I9r3Mc8Ol0jYtz0yNQDSzVLrGCaJlC311trU70tZ+ZkAVV6Mn8lOhSbj1cK0lvSr6ZK4dgqGl3I1eTZJJhbLNdg7UOVaiRx9543+C/p/As7w== brjackma@redhat.com" 2026-03-09 18:21:38.263731 | crc | } 2026-03-09 18:21:38.628435 | crc | changed: 2026-03-09 18:21:38.628587 | crc | { 2026-03-09 18:21:38.628616 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwedoZ0TWPJX/z/4TAbO/kKcDZOQVgRH0hAqrL5UCI1 vcastell@redhat.com" 2026-03-09 18:21:38.628635 | crc | } 2026-03-09 18:21:39.037137 | crc | changed: 2026-03-09 18:21:39.037301 | crc | { 2026-03-09 18:21:39.037360 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmv8sE8GCk6ZTPIqF0FQrttBdL3mq7rCm/IJy0xDFh7 michburk@redhat.com" 2026-03-09 18:21:39.037546 | crc | } 2026-03-09 18:21:39.412926 | crc | changed: 2026-03-09 18:21:39.413055 | crc | { 2026-03-09 18:21:39.413084 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICy6GpGEtwevXEEn4mmLR5lmSLe23dGgAvzkB9DMNbkf rsafrono@rsafrono" 2026-03-09 18:21:39.413104 | crc | } 2026-03-09 18:21:39.441528 | 2026-03-09 18:21:39.441650 | TASK [Set timezone to UTC] 2026-03-09 18:21:40.007564 | controller | changed: executed `/bin/timedatectl set-timezone UTC` 2026-03-09 18:21:40.097196 | crc | ok 2026-03-09 18:21:40.103172 | 2026-03-09 18:21:40.103241 | TASK [Create nodepool directory] 2026-03-09 18:21:40.310228 | controller | changed 2026-03-09 18:21:40.420481 | crc | changed 2026-03-09 18:21:40.426899 | 2026-03-09 18:21:40.426980 | TASK [Create nodepool sub_nodes file] 2026-03-09 18:21:41.162647 | controller | changed 2026-03-09 18:21:41.219292 | crc | changed 2026-03-09 18:21:41.226999 | 2026-03-09 18:21:41.227126 | TASK [Create nodepool sub_nodes_private file] 2026-03-09 18:21:41.874276 | controller | changed 2026-03-09 18:21:42.134389 | crc | changed 2026-03-09 18:21:42.148966 | 2026-03-09 18:21:42.149137 | LOOP [Populate nodepool sub_nodes file] 2026-03-09 18:21:42.204389 | 2026-03-09 18:21:42.204570 | LOOP [Populate nodepool sub_nodes_private file] 2026-03-09 18:21:42.250300 | 2026-03-09 18:21:42.250498 | TASK [Create nodepool primary file] 2026-03-09 18:21:42.278075 | controller | skipping: Conditional result was False 2026-03-09 18:21:42.289296 | crc | skipping: Conditional result was False 2026-03-09 18:21:42.296859 | 2026-03-09 18:21:42.297032 | TASK [Create nodepool node_private for this node] 2026-03-09 18:21:42.993427 | controller | changed 2026-03-09 18:21:43.229103 | crc | changed 2026-03-09 18:21:43.244524 | 2026-03-09 18:21:43.244701 | LOOP [Copy ssh keys to nodepool directory] 2026-03-09 18:21:43.674420 | controller | ok: Item: id_rsa Runtime: 0:00:00.005794 2026-03-09 18:21:43.674636 | 2026-03-09 18:21:43.698910 | crc | ok: Item: id_rsa Runtime: 0:00:00.009405 2026-03-09 18:21:43.882400 | controller | ok: Item: id_rsa.pub Runtime: 0:00:00.005646 2026-03-09 18:21:43.928340 | crc | ok: Item: id_rsa.pub Runtime: 0:00:00.006793 2026-03-09 18:21:43.945755 | 2026-03-09 18:21:43.945978 | TASK [Add sudoers role for zuul-sudo-grep.sh] 2026-03-09 18:21:44.603123 | controller | changed 2026-03-09 18:21:44.844777 | crc | changed 2026-03-09 18:21:44.856886 | 2026-03-09 18:21:44.857044 | TASK [Validate sudoers config after edits] 2026-03-09 18:21:45.161995 | controller | /etc/sudoers: parsed OK 2026-03-09 18:21:45.162244 | controller | /etc/sudoers.d/90-cloud-init-users: parsed OK 2026-03-09 18:21:45.162368 | controller | /etc/sudoers.d/zuul: parsed OK 2026-03-09 18:21:45.162378 | controller | /etc/sudoers.d/zuul-sudo-grep: parsed OK 2026-03-09 18:21:45.245323 | crc | /etc/sudoers: parsed OK 2026-03-09 18:21:45.245536 | crc | /etc/sudoers.d/coreos-sudo-group: parsed OK 2026-03-09 18:21:45.245554 | crc | /etc/sudoers.d/zuul-sudo-grep: parsed OK 2026-03-09 18:21:45.430874 | controller | ok: Runtime: 0:00:00.007226 2026-03-09 18:21:45.449473 | crc | ok: Runtime: 0:00:00.011371 2026-03-09 18:21:45.464155 | 2026-03-09 18:21:45.464327 | TASK [Show the environment passed in to job shell scripts] 2026-03-09 18:21:45.754428 | controller | SHELL=/bin/bash 2026-03-09 18:21:45.755222 | controller | ZUUL_PROJECT=openstack-k8s-operators/watcher-operator 2026-03-09 18:21:45.755264 | controller | ZUUL_SHORT_PROJECT_NAME=watcher-operator 2026-03-09 18:21:45.755271 | controller | ZUUL_CHANGES=openstack/watcher:master:refs/changes/76/979276/3^openstack/watcher:master:refs/changes/43/979643/3^openstack/watcher-tempest-plugin:master:refs/changes/44/958644/32^openstack/watcher-tempest-plugin:master:refs/changes/02/962702/25^openstack/watcher-tempest-plugin:master:refs/changes/59/963559/16^openstack-k8s-operators/watcher-operator:main:refs/changes/51/351/2e035aad6e396aeb72cc6aec8684c43e59f8b674 2026-03-09 18:21:45.755277 | controller | PWD=/home/zuul 2026-03-09 18:21:45.755282 | controller | ZUUL_PIPELINE=github-check 2026-03-09 18:21:45.755286 | controller | LOGNAME=zuul 2026-03-09 18:21:45.755290 | controller | XDG_SESSION_TYPE=tty 2026-03-09 18:21:45.755295 | controller | _=/usr/bin/env 2026-03-09 18:21:45.755300 | controller | MOTD_SHOWN=pam 2026-03-09 18:21:45.755304 | controller | HOME=/home/zuul 2026-03-09 18:21:45.755308 | controller | LANG=en_US.UTF-8 2026-03-09 18:21:45.755313 | controller | SSH_CONNECTION=38.102.83.114 51684 38.102.83.2 22 2026-03-09 18:21:45.755318 | controller | TOX_TESTENV_PASSENV=ZUUL_UUID ZUUL_PROJECT ZUUL_SHORT_PROJECT_NAME ZUUL_PIPELINE ZUUL_VOTING WORKSPACE BUILD_TIMEOUT ZUUL_BRANCH ZUUL_CHANGES ZUUL_CHANGE_IDS ZUUL_CHANGE ZUUL_PATCHSET 2026-03-09 18:21:45.755326 | controller | ZUUL_CHANGE_IDS=979276,3 979643,3 958644,32 962702,25 963559,16 351,2e035aad6e396aeb72cc6aec8684c43e59f8b674 2026-03-09 18:21:45.755331 | controller | WORKSPACE=/home/zuul/workspace 2026-03-09 18:21:45.755336 | controller | XDG_SESSION_CLASS=user 2026-03-09 18:21:45.755341 | controller | SELINUX_ROLE_REQUESTED= 2026-03-09 18:21:45.755345 | controller | LESSOPEN=||/usr/bin/lesspipe.sh %s 2026-03-09 18:21:45.755351 | controller | USER=zuul 2026-03-09 18:21:45.755381 | controller | ZUUL_VOTING=True 2026-03-09 18:21:45.755387 | controller | BUILD_TIMEOUT=7200000 2026-03-09 18:21:45.755392 | controller | SELINUX_USE_CURRENT_RANGE= 2026-03-09 18:21:45.755397 | controller | SHLVL=1 2026-03-09 18:21:45.755403 | controller | ZUUL_PATCHSET=2e035aad6e396aeb72cc6aec8684c43e59f8b674 2026-03-09 18:21:45.755408 | controller | XDG_SESSION_ID=1 2026-03-09 18:21:45.755413 | controller | ZUUL_BRANCH=main 2026-03-09 18:21:45.755418 | controller | XDG_RUNTIME_DIR=/run/user/1000 2026-03-09 18:21:45.755423 | controller | SSH_CLIENT=38.102.83.114 51684 22 2026-03-09 18:21:45.755429 | controller | DEBUGINFOD_URLS=https://debuginfod.centos.org/ 2026-03-09 18:21:45.755434 | controller | DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima: 2026-03-09 18:21:45.755439 | controller | which_declare=declare -f 2026-03-09 18:21:45.755445 | controller | PATH=/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin 2026-03-09 18:21:45.755451 | controller | SELINUX_LEVEL_REQUESTED= 2026-03-09 18:21:45.755456 | controller | ZUUL_CHANGE=351 2026-03-09 18:21:45.755461 | controller | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus 2026-03-09 18:21:45.755466 | controller | ZUUL_UUID=e72ba1c8eae94a9da88ed53f3a62dcb0 2026-03-09 18:21:45.755471 | controller | BASH_FUNC_which%%=() { ( alias; 2026-03-09 18:21:45.755477 | controller | eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@ 2026-03-09 18:21:45.755482 | controller | } 2026-03-09 18:21:45.866760 | crc | SHELL=/bin/bash 2026-03-09 18:21:45.866858 | crc | ZUUL_PROJECT=openstack-k8s-operators/watcher-operator 2026-03-09 18:21:45.866870 | crc | ZUUL_SHORT_PROJECT_NAME=watcher-operator 2026-03-09 18:21:45.866879 | crc | ZUUL_CHANGES=openstack/watcher:master:refs/changes/76/979276/3^openstack/watcher:master:refs/changes/43/979643/3^openstack/watcher-tempest-plugin:master:refs/changes/44/958644/32^openstack/watcher-tempest-plugin:master:refs/changes/02/962702/25^openstack/watcher-tempest-plugin:master:refs/changes/59/963559/16^openstack-k8s-operators/watcher-operator:main:refs/changes/51/351/2e035aad6e396aeb72cc6aec8684c43e59f8b674 2026-03-09 18:21:45.866892 | crc | PWD=/var/home/core 2026-03-09 18:21:45.866900 | crc | ZUUL_PIPELINE=github-check 2026-03-09 18:21:45.866907 | crc | LOGNAME=core 2026-03-09 18:21:45.866914 | crc | XDG_SESSION_TYPE=tty 2026-03-09 18:21:45.866921 | crc | _=/usr/bin/env 2026-03-09 18:21:45.866927 | crc | MOTD_SHOWN=pam 2026-03-09 18:21:45.866933 | crc | HOME=/var/home/core 2026-03-09 18:21:45.866940 | crc | LANG=C.UTF-8 2026-03-09 18:21:45.866946 | crc | SSH_CONNECTION=38.102.83.114 33290 38.102.83.74 22 2026-03-09 18:21:45.866953 | crc | TOX_TESTENV_PASSENV=ZUUL_UUID ZUUL_PROJECT ZUUL_SHORT_PROJECT_NAME ZUUL_PIPELINE ZUUL_VOTING WORKSPACE BUILD_TIMEOUT ZUUL_BRANCH ZUUL_CHANGES ZUUL_CHANGE_IDS ZUUL_CHANGE ZUUL_PATCHSET 2026-03-09 18:21:45.866964 | crc | ZUUL_CHANGE_IDS=979276,3 979643,3 958644,32 962702,25 963559,16 351,2e035aad6e396aeb72cc6aec8684c43e59f8b674 2026-03-09 18:21:45.866970 | crc | WORKSPACE=/home/zuul/workspace 2026-03-09 18:21:45.866977 | crc | XDG_SESSION_CLASS=user 2026-03-09 18:21:45.866984 | crc | SELINUX_ROLE_REQUESTED= 2026-03-09 18:21:45.866991 | crc | LESSOPEN=||/usr/bin/lesspipe.sh %s 2026-03-09 18:21:45.866997 | crc | USER=core 2026-03-09 18:21:45.867004 | crc | ZUUL_VOTING=True 2026-03-09 18:21:45.867010 | crc | BUILD_TIMEOUT=7200000 2026-03-09 18:21:45.867016 | crc | SELINUX_USE_CURRENT_RANGE= 2026-03-09 18:21:45.867023 | crc | SHLVL=1 2026-03-09 18:21:45.867030 | crc | ZUUL_PATCHSET=2e035aad6e396aeb72cc6aec8684c43e59f8b674 2026-03-09 18:21:45.867037 | crc | XDG_SESSION_ID=2 2026-03-09 18:21:45.867140 | crc | ZUUL_BRANCH=main 2026-03-09 18:21:45.867149 | crc | XDG_RUNTIME_DIR=/run/user/1000 2026-03-09 18:21:45.867156 | crc | SSH_CLIENT=38.102.83.114 33290 22 2026-03-09 18:21:45.867162 | crc | which_declare=declare -f 2026-03-09 18:21:45.867169 | crc | PATH=/var/home/core/.local/bin:/var/home/core/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin 2026-03-09 18:21:45.867176 | crc | SELINUX_LEVEL_REQUESTED= 2026-03-09 18:21:45.867205 | crc | ZUUL_CHANGE=351 2026-03-09 18:21:45.867212 | crc | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus 2026-03-09 18:21:45.867218 | crc | ZUUL_UUID=e72ba1c8eae94a9da88ed53f3a62dcb0 2026-03-09 18:21:45.867225 | crc | BASH_FUNC_which%%=() { ( alias; 2026-03-09 18:21:45.867234 | crc | eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@ 2026-03-09 18:21:45.867240 | crc | } 2026-03-09 18:21:46.052305 | controller | ok: Runtime: 0:00:00.006919 2026-03-09 18:21:46.074002 | crc | ok: Runtime: 0:00:00.013421 2026-03-09 18:21:46.087594 | 2026-03-09 18:21:46.087735 | TASK [Workaround hardcoded /home/zuul/workspace in zuul_legacy_vars] 2026-03-09 18:21:46.111388 | controller | skipping: Conditional result was False 2026-03-09 18:21:46.132993 | crc | skipping: Conditional result was False 2026-03-09 18:21:46.143489 | 2026-03-09 18:21:46.143592 | TASK [Symlink /home/zuul-worker/workspace] 2026-03-09 18:21:46.733273 | controller | skipping: Conditional result was False 2026-03-09 18:21:46.734289 | crc | skipping: Conditional result was False 2026-03-09 18:21:46.746424 | 2026-03-09 18:21:46.746560 | TASK [Ensure legacy workspace directory] 2026-03-09 18:21:47.061645 | controller | changed 2026-03-09 18:21:47.069680 | crc | changed 2026-03-09 18:21:47.115095 | 2026-03-09 18:21:47.115197 | PLAY RECAP 2026-03-09 18:21:47.115248 | controller | ok: 14 changed: 11 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-09 18:21:47.115279 | crc | ok: 14 changed: 10 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-09 18:21:47.115301 | 2026-03-09 18:21:47.240812 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/pre.yaml@master] 2026-03-09 18:21:47.245432 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network.yml@master] 2026-03-09 18:21:47.931822 | 2026-03-09 18:21:47.931966 | PLAY [controller] 2026-03-09 18:21:47.960127 | 2026-03-09 18:21:47.960270 | TASK [Debug parameters] 2026-03-09 18:21:47.990630 | controller | ok 2026-03-09 18:21:47.997382 | 2026-03-09 18:21:47.997516 | TASK [Assert crc_ci_bootstrap_cloud_name is defined] 2026-03-09 18:21:48.027695 | controller | ok: All assertions passed 2026-03-09 18:21:48.032828 | 2026-03-09 18:21:48.032937 | TASK [Assert crc_ci_bootstrap_networking is defined and contains the minimum fields] 2026-03-09 18:21:48.073736 | controller | ok: All assertions passed 2026-03-09 18:21:48.080063 | 2026-03-09 18:21:48.080158 | TASK [Assert that the default network exists and has no VLAN tag] 2026-03-09 18:21:48.113122 | controller | ok: All assertions passed 2026-03-09 18:21:48.120544 | 2026-03-09 18:21:48.120658 | LOOP [Assert that all instances has a given fixed IP for each net and default net is present] 2026-03-09 18:21:48.235073 | controller | ok: All assertions passed 2026-03-09 18:21:48.258574 | controller | ok: All assertions passed 2026-03-09 18:21:48.268174 | 2026-03-09 18:21:48.268301 | LOOP [Assert that every host references an valid/declared network] 2026-03-09 18:21:48.346718 | controller | ok: All assertions passed 2026-03-09 18:21:48.346943 | controller | ok: All items complete 2026-03-09 18:21:48.346972 | 2026-03-09 18:21:48.358657 | controller | ok: All assertions passed 2026-03-09 18:21:48.374729 | 2026-03-09 18:21:48.374892 | LOOP [Assert that all instances has in range fixed IP for each net] 2026-03-09 18:21:48.484236 | controller | ok: All assertions passed 2026-03-09 18:21:48.509159 | controller | ok: All assertions passed 2026-03-09 18:21:48.536897 | 2026-03-09 18:21:48.537240 | TASK [Assert that all non-default nets has the vlan tag] 2026-03-09 18:21:48.625575 | controller | ok: All assertions passed 2026-03-09 18:21:48.634195 | 2026-03-09 18:21:48.634353 | TASK [Install required packages] 2026-03-09 18:22:03.477333 | controller | changed 2026-03-09 18:22:03.492994 | 2026-03-09 18:22:03.493193 | TASK [Create openstack config dir] 2026-03-09 18:22:03.873665 | controller | changed 2026-03-09 18:22:03.880756 | 2026-03-09 18:22:03.880829 | LOOP [Create ci dir in etc folder in the controller] 2026-03-09 18:22:04.202303 | controller -> 38.102.83.74 | changed: "crc" 2026-03-09 18:22:04.202564 | controller -> 38.102.83.74 | ok: All items complete 2026-03-09 18:22:04.202594 | 2026-03-09 18:22:04.458158 | controller -> 38.102.83.2 | changed: "controller" 2026-03-09 18:22:04.470377 | 2026-03-09 18:22:04.474669 | TASK [Set global resource names as facts] 2026-03-09 18:22:04.530276 | controller | ok 2026-03-09 18:22:04.540743 | 2026-03-09 18:22:04.540888 | TASK [Generate CA cert file] 2026-03-09 18:22:04.587982 | controller | skipping: Conditional result was False 2026-03-09 18:22:04.604484 | 2026-03-09 18:22:04.604636 | TASK [Update CA certs] 2026-03-09 18:22:05.166412 | controller | skipping: Conditional result was False 2026-03-09 18:22:05.180696 | 2026-03-09 18:22:05.180879 | TASK [Generate clouds config from cloud_secrets secret] 2026-03-09 18:22:06.140704 | controller | changed 2026-03-09 18:22:06.165271 | 2026-03-09 18:22:06.165415 | TASK [Create the default network] 2026-03-09 18:22:08.869632 | controller | admin_state_up: true 2026-03-09 18:22:08.869713 | controller | availability_zone_hints: 2026-03-09 18:22:08.869798 | controller | - nova 2026-03-09 18:22:08.869807 | controller | availability_zones: [] 2026-03-09 18:22:08.870006 | controller | created_at: '2026-03-09T18:22:07Z' 2026-03-09 18:22:08.870109 | controller | description: '' 2026-03-09 18:22:08.870214 | controller | dns_domain: '' 2026-03-09 18:22:08.870323 | controller | id: d09b02e2-af90-41f7-a8a9-5064393c43d0 2026-03-09 18:22:08.870454 | controller | ipv4_address_scope: null 2026-03-09 18:22:08.870551 | controller | ipv6_address_scope: null 2026-03-09 18:22:08.870670 | controller | is_default: false 2026-03-09 18:22:08.870763 | controller | is_vlan_qinq: null 2026-03-09 18:22:08.870860 | controller | is_vlan_transparent: true 2026-03-09 18:22:08.870951 | controller | l2_adjacency: true 2026-03-09 18:22:08.871070 | controller | mtu: 1500 2026-03-09 18:22:08.871180 | controller | name: zuul-ci-net-e72ba1c8 2026-03-09 18:22:08.871283 | controller | port_security_enabled: false 2026-03-09 18:22:08.871392 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-09 18:22:08.871492 | controller | provider:network_type: null 2026-03-09 18:22:08.871594 | controller | provider:physical_network: null 2026-03-09 18:22:08.871698 | controller | provider:segmentation_id: null 2026-03-09 18:22:08.871806 | controller | qos_policy_id: null 2026-03-09 18:22:08.871901 | controller | revision_number: 1 2026-03-09 18:22:08.871996 | controller | router:external: false 2026-03-09 18:22:08.872087 | controller | segments: null 2026-03-09 18:22:08.872229 | controller | shared: false 2026-03-09 18:22:08.872321 | controller | status: ACTIVE 2026-03-09 18:22:08.872406 | controller | subnets: [] 2026-03-09 18:22:08.872490 | controller | tags: [] 2026-03-09 18:22:08.872598 | controller | updated_at: '2026-03-09T18:22:08Z' 2026-03-09 18:22:09.209888 | controller | changed 2026-03-09 18:22:09.216129 | 2026-03-09 18:22:09.216225 | TASK [Create subnet - ipv4] 2026-03-09 18:22:12.281657 | controller | allocation_pools: 2026-03-09 18:22:12.281740 | controller | - end: 192.168.122.254 2026-03-09 18:22:12.281747 | controller | start: 192.168.122.2 2026-03-09 18:22:12.281754 | controller | cidr: 192.168.122.0/24 2026-03-09 18:22:12.281854 | controller | created_at: '2026-03-09T18:22:12Z' 2026-03-09 18:22:12.281956 | controller | description: '' 2026-03-09 18:22:12.282050 | controller | dns_nameservers: [] 2026-03-09 18:22:12.282183 | controller | dns_publish_fixed_ip: null 2026-03-09 18:22:12.282308 | controller | enable_dhcp: false 2026-03-09 18:22:12.282404 | controller | gateway_ip: 192.168.122.1 2026-03-09 18:22:12.282496 | controller | host_routes: [] 2026-03-09 18:22:12.282605 | controller | id: edcfb140-8934-4cc7-a46a-d8cf20ecfdf7 2026-03-09 18:22:12.282722 | controller | ip_version: 4 2026-03-09 18:22:12.282820 | controller | ipv6_address_mode: null 2026-03-09 18:22:12.282915 | controller | ipv6_ra_mode: null 2026-03-09 18:22:12.283017 | controller | name: zuul-ci-subnet-e72ba1c8 2026-03-09 18:22:12.283132 | controller | network_id: d09b02e2-af90-41f7-a8a9-5064393c43d0 2026-03-09 18:22:12.283259 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-09 18:22:12.283354 | controller | revision_number: 0 2026-03-09 18:22:12.283447 | controller | segment_id: null 2026-03-09 18:22:12.283541 | controller | service_types: [] 2026-03-09 18:22:12.283636 | controller | subnetpool_id: null 2026-03-09 18:22:12.283722 | controller | tags: [] 2026-03-09 18:22:12.283831 | controller | updated_at: '2026-03-09T18:22:12Z' 2026-03-09 18:22:12.750243 | controller | changed 2026-03-09 18:22:12.756172 | 2026-03-09 18:22:12.756286 | TASK [Set yaml returned data for further usage] 2026-03-09 18:22:12.837588 | controller | ok 2026-03-09 18:22:12.844045 | 2026-03-09 18:22:12.844163 | TASK [Create router] 2026-03-09 18:22:13.413049 | controller | skipping: Conditional result was False 2026-03-09 18:22:13.438257 | 2026-03-09 18:22:13.438375 | TASK [Set router yaml returned data for further usage] 2026-03-09 18:22:13.494133 | controller | skipping: Conditional result was False 2026-03-09 18:22:13.499564 | 2026-03-09 18:22:13.499629 | TASK [Add router to subnet] 2026-03-09 18:22:14.034078 | controller | skipping: Conditional result was False 2026-03-09 18:22:14.040979 | 2026-03-09 18:22:14.041111 | LOOP [Create ports, VLANs and configure them on the instance] 2026-03-09 18:22:14.097600 | controller | ok: 2026-03-09 18:22:14.097843 | controller | { 2026-03-09 18:22:14.097876 | controller | "key": "controller", 2026-03-09 18:22:14.097901 | controller | "value": { 2026-03-09 18:22:14.097924 | controller | "networks": { 2026-03-09 18:22:14.097947 | controller | "default": { 2026-03-09 18:22:14.097969 | controller | "ip": "192.168.122.11" 2026-03-09 18:22:14.097990 | controller | } 2026-03-09 18:22:14.098037 | controller | } 2026-03-09 18:22:14.098066 | controller | } 2026-03-09 18:22:14.098089 | controller | } 2026-03-09 18:22:17.136299 | controller | ok: 2026-03-09 18:22:17.136412 | controller | { 2026-03-09 18:22:17.136446 | controller | "key": "crc", 2026-03-09 18:22:17.136473 | controller | "value": { 2026-03-09 18:22:17.136497 | controller | "networks": { 2026-03-09 18:22:17.136522 | controller | "default": { 2026-03-09 18:22:17.136546 | controller | "ip": "192.168.122.10" 2026-03-09 18:22:17.136571 | controller | }, 2026-03-09 18:22:17.136596 | controller | "internal-api": { 2026-03-09 18:22:17.136619 | controller | "ip": "172.17.0.5" 2026-03-09 18:22:17.136652 | controller | }, 2026-03-09 18:22:17.136675 | controller | "storage": { 2026-03-09 18:22:17.136698 | controller | "ip": "172.18.0.5" 2026-03-09 18:22:17.136721 | controller | }, 2026-03-09 18:22:17.136743 | controller | "tenant": { 2026-03-09 18:22:17.136766 | controller | "ip": "172.19.0.5" 2026-03-09 18:22:17.136789 | controller | } 2026-03-09 18:22:17.136812 | controller | } 2026-03-09 18:22:17.136834 | controller | } 2026-03-09 18:22:17.136856 | controller | } 2026-03-09 18:22:17.150369 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance.yml 2026-03-09 18:22:17.155328 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance.yml 2026-03-09 18:22:17.166759 | 2026-03-09 18:22:17.166864 | TASK [Create instance ports and attach if necessary] 2026-03-09 18:22:17.190383 | controller | ok 2026-03-09 18:22:17.250310 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance-create-attach-port.yml 2026-03-09 18:22:17.257192 | 2026-03-09 18:22:17.257272 | TASK [Print the host networking data] 2026-03-09 18:22:17.309901 | controller | ok 2026-03-09 18:22:17.317444 | 2026-03-09 18:22:17.317532 | TASK [Create the private network parent port] 2026-03-09 18:22:20.248745 | controller | admin_state_up: true 2026-03-09 18:22:20.249025 | controller | allowed_address_pairs: [] 2026-03-09 18:22:20.249301 | controller | binding_host_id: null 2026-03-09 18:22:20.249477 | controller | binding_profile: {} 2026-03-09 18:22:20.249650 | controller | binding_vif_details: {} 2026-03-09 18:22:20.249841 | controller | binding_vif_type: null 2026-03-09 18:22:20.250049 | controller | binding_vnic_type: normal 2026-03-09 18:22:20.250247 | controller | created_at: '2026-03-09T18:22:19Z' 2026-03-09 18:22:20.250416 | controller | data_plane_status: null 2026-03-09 18:22:20.250566 | controller | description: '' 2026-03-09 18:22:20.250721 | controller | device_id: '' 2026-03-09 18:22:20.250867 | controller | device_owner: '' 2026-03-09 18:22:20.251012 | controller | device_profile: null 2026-03-09 18:22:20.251409 | controller | dns_assignment: 2026-03-09 18:22:20.251563 | controller | - fqdn: host-192-168-122-11.openstacklocal. 2026-03-09 18:22:20.251575 | controller | hostname: host-192-168-122-11 2026-03-09 18:22:20.251582 | controller | ip_address: 192.168.122.11 2026-03-09 18:22:20.251590 | controller | dns_domain: '' 2026-03-09 18:22:20.251721 | controller | dns_name: '' 2026-03-09 18:22:20.251859 | controller | extra_dhcp_opts: [] 2026-03-09 18:22:20.252210 | controller | fixed_ips: 2026-03-09 18:22:20.252398 | controller | - ip_address: 192.168.122.11 2026-03-09 18:22:20.252412 | controller | subnet_id: edcfb140-8934-4cc7-a46a-d8cf20ecfdf7 2026-03-09 18:22:20.252422 | controller | hardware_offload_type: null 2026-03-09 18:22:20.252580 | controller | hints: '' 2026-03-09 18:22:20.252781 | controller | id: 6d648f67-f72a-4d32-8184-c2d17ffb3d8c 2026-03-09 18:22:20.252956 | controller | ip_allocation: immediate 2026-03-09 18:22:20.253141 | controller | mac_address: fa:16:3e:08:be:f9 2026-03-09 18:22:20.253376 | controller | name: controller-593e5d4e-760b-40a9-aa62-62e7bf2adfde 2026-03-09 18:22:20.253585 | controller | network_id: d09b02e2-af90-41f7-a8a9-5064393c43d0 2026-03-09 18:22:20.253770 | controller | numa_affinity_policy: null 2026-03-09 18:22:20.254023 | controller | port_security_enabled: false 2026-03-09 18:22:20.254288 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-09 18:22:20.254480 | controller | propagate_uplink_status: null 2026-03-09 18:22:20.254658 | controller | resource_request: null 2026-03-09 18:22:20.254874 | controller | revision_number: 1 2026-03-09 18:22:20.255060 | controller | qos_network_policy_id: null 2026-03-09 18:22:20.255250 | controller | qos_policy_id: null 2026-03-09 18:22:20.255432 | controller | security_group_ids: [] 2026-03-09 18:22:20.255601 | controller | status: DOWN 2026-03-09 18:22:20.255757 | controller | tags: [] 2026-03-09 18:22:20.255926 | controller | trunk_details: null 2026-03-09 18:22:20.256094 | controller | trusted: null 2026-03-09 18:22:20.256305 | controller | updated_at: '2026-03-09T18:22:19Z' 2026-03-09 18:22:20.876485 | controller | changed 2026-03-09 18:22:20.891231 | 2026-03-09 18:22:20.891478 | TASK [Set common facts for further usage] 2026-03-09 18:22:20.986916 | controller | ok 2026-03-09 18:22:20.999274 | 2026-03-09 18:22:20.999473 | TASK [Create the trunk port] 2026-03-09 18:22:21.547827 | controller | skipping: Conditional result was False 2026-03-09 18:22:21.554799 | 2026-03-09 18:22:21.554896 | TASK [Avoid NetworkManager to automatically picked up the new interface] 2026-03-09 18:22:21.609648 | controller | skipping: Conditional result was False 2026-03-09 18:22:21.617314 | 2026-03-09 18:22:21.617420 | TASK [Refresh NetworkManager] 2026-03-09 18:22:21.682985 | controller | skipping: Conditional result was False 2026-03-09 18:22:21.690218 | 2026-03-09 18:22:21.690320 | TASK [Attach the port to the instance] 2026-03-09 18:22:28.764750 | controller | changed 2026-03-09 18:22:28.774179 | 2026-03-09 18:22:28.774294 | TASK [Wait till the interface exists] 2026-03-09 18:22:29.080303 | controller -> 38.102.83.2 | changed 2026-03-09 18:22:29.122989 | 2026-03-09 18:22:29.123104 | TASK [Prepare interface data to be used configuring trunk and vlan ifaces] 2026-03-09 18:22:29.280260 | controller | ok 2026-03-09 18:22:29.286480 | 2026-03-09 18:22:29.286587 | LOOP [Create VLANs ports and attach them to the trunk] 2026-03-09 18:22:29.310238 | 2026-03-09 18:22:29.310417 | LOOP [Prepare VLANs interface data to be used configuring] 2026-03-09 18:22:29.333600 | 2026-03-09 18:22:29.333785 | LOOP [Create NetworkManager configuration file for the trunk port] 2026-03-09 18:22:30.014708 | controller -> 38.102.83.2 | changed: 2026-03-09 18:22:30.015123 | controller -> 38.102.83.2 | { 2026-03-09 18:22:30.015182 | controller -> 38.102.83.2 | "key": "default", 2026-03-09 18:22:30.015221 | controller -> 38.102.83.2 | "value": { 2026-03-09 18:22:30.015257 | controller -> 38.102.83.2 | "connection": "ci-private-network", 2026-03-09 18:22:30.015292 | controller -> 38.102.83.2 | "gw": "192.168.122.1", 2026-03-09 18:22:30.015330 | controller -> 38.102.83.2 | "iface": "eth1", 2026-03-09 18:22:30.015364 | controller -> 38.102.83.2 | "ip": "192.168.122.11/24", 2026-03-09 18:22:30.015397 | controller -> 38.102.83.2 | "mac": "fa:16:3e:08:be:f9", 2026-03-09 18:22:30.015429 | controller -> 38.102.83.2 | "mtu": "1500" 2026-03-09 18:22:30.015461 | controller -> 38.102.83.2 | } 2026-03-09 18:22:30.015492 | controller -> 38.102.83.2 | } 2026-03-09 18:22:30.015553 | 2026-03-09 18:22:30.034788 | 2026-03-09 18:22:30.034986 | TASK [Refresh NetworkManager] 2026-03-09 18:22:30.831724 | controller -> 38.102.83.2 | changed 2026-03-09 18:22:30.842204 | 2026-03-09 18:22:30.842340 | TASK [Debug fetch IP routes] 2026-03-09 18:22:31.143824 | controller -> 38.102.83.2 | changed 2026-03-09 18:22:31.159954 | 2026-03-09 18:22:31.160132 | TASK [Debug IP routes] 2026-03-09 18:22:31.196955 | controller -> 38.102.83.2 | ok 2026-03-09 18:22:31.205675 | 2026-03-09 18:22:31.205797 | TASK [Create instance ports and attach if necessary] 2026-03-09 18:22:31.233178 | controller | ok 2026-03-09 18:22:31.253520 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance-create-attach-port.yml 2026-03-09 18:22:31.261963 | 2026-03-09 18:22:31.262066 | TASK [Print the host networking data] 2026-03-09 18:22:31.295599 | controller | ok 2026-03-09 18:22:31.303098 | 2026-03-09 18:22:31.303217 | TASK [Create the private network parent port] 2026-03-09 18:22:34.767660 | controller | admin_state_up: true 2026-03-09 18:22:34.767834 | controller | allowed_address_pairs: [] 2026-03-09 18:22:34.768100 | controller | binding_host_id: null 2026-03-09 18:22:34.768320 | controller | binding_profile: {} 2026-03-09 18:22:34.768512 | controller | binding_vif_details: {} 2026-03-09 18:22:34.768714 | controller | binding_vif_type: null 2026-03-09 18:22:34.768969 | controller | binding_vnic_type: normal 2026-03-09 18:22:34.769273 | controller | created_at: '2026-03-09T18:22:33Z' 2026-03-09 18:22:34.769475 | controller | data_plane_status: null 2026-03-09 18:22:34.769671 | controller | description: '' 2026-03-09 18:22:34.769842 | controller | device_id: '' 2026-03-09 18:22:34.770013 | controller | device_owner: '' 2026-03-09 18:22:34.770231 | controller | device_profile: null 2026-03-09 18:22:34.770791 | controller | dns_assignment: 2026-03-09 18:22:34.770972 | controller | - fqdn: host-192-168-122-10.openstacklocal. 2026-03-09 18:22:34.770979 | controller | hostname: host-192-168-122-10 2026-03-09 18:22:34.770984 | controller | ip_address: 192.168.122.10 2026-03-09 18:22:34.770990 | controller | dns_domain: '' 2026-03-09 18:22:34.771158 | controller | dns_name: '' 2026-03-09 18:22:34.771349 | controller | extra_dhcp_opts: [] 2026-03-09 18:22:34.771786 | controller | fixed_ips: 2026-03-09 18:22:34.771978 | controller | - ip_address: 192.168.122.10 2026-03-09 18:22:34.771985 | controller | subnet_id: edcfb140-8934-4cc7-a46a-d8cf20ecfdf7 2026-03-09 18:22:34.771991 | controller | hardware_offload_type: null 2026-03-09 18:22:34.772100 | controller | hints: '' 2026-03-09 18:22:34.772269 | controller | id: 2a0fed40-01ed-4d11-863e-67f7d40991d7 2026-03-09 18:22:34.772402 | controller | ip_allocation: immediate 2026-03-09 18:22:34.772568 | controller | mac_address: fa:16:3e:d3:67:c2 2026-03-09 18:22:34.772821 | controller | name: crc-ea3e2df3-251a-4cdb-9064-3c52ac509aba 2026-03-09 18:22:34.773065 | controller | network_id: d09b02e2-af90-41f7-a8a9-5064393c43d0 2026-03-09 18:22:34.773279 | controller | numa_affinity_policy: null 2026-03-09 18:22:34.773470 | controller | port_security_enabled: false 2026-03-09 18:22:34.773656 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-09 18:22:34.773807 | controller | propagate_uplink_status: null 2026-03-09 18:22:34.773950 | controller | resource_request: null 2026-03-09 18:22:34.774120 | controller | revision_number: 1 2026-03-09 18:22:34.774275 | controller | qos_network_policy_id: null 2026-03-09 18:22:34.774404 | controller | qos_policy_id: null 2026-03-09 18:22:34.774552 | controller | security_group_ids: [] 2026-03-09 18:22:34.774701 | controller | status: DOWN 2026-03-09 18:22:34.774855 | controller | tags: [] 2026-03-09 18:22:34.775035 | controller | trunk_details: null 2026-03-09 18:22:34.775216 | controller | trusted: null 2026-03-09 18:22:34.775384 | controller | updated_at: '2026-03-09T18:22:33Z' 2026-03-09 18:22:35.341582 | controller | changed 2026-03-09 18:22:35.360209 | 2026-03-09 18:22:35.360414 | TASK [Set common facts for further usage] 2026-03-09 18:22:35.457317 | controller | ok 2026-03-09 18:22:35.464314 | 2026-03-09 18:22:35.464391 | TASK [Create the trunk port] 2026-03-09 18:22:35.996950 | controller | skipping: Conditional result was False 2026-03-09 18:22:36.007340 | 2026-03-09 18:22:36.007530 | TASK [Avoid NetworkManager to automatically picked up the new interface] 2026-03-09 18:22:36.065331 | controller | skipping: Conditional result was False 2026-03-09 18:22:36.075649 | 2026-03-09 18:22:36.075797 | TASK [Refresh NetworkManager] 2026-03-09 18:22:36.133924 | controller | skipping: Conditional result was False 2026-03-09 18:22:36.144143 | 2026-03-09 18:22:36.144302 | TASK [Attach the port to the instance] 2026-03-09 18:22:44.714803 | controller | changed 2026-03-09 18:22:44.723566 | 2026-03-09 18:22:44.723697 | TASK [Wait till the interface exists] 2026-03-09 18:22:45.099042 | controller -> 38.102.83.74 | changed 2026-03-09 18:22:45.106264 | 2026-03-09 18:22:45.106386 | TASK [Prepare interface data to be used configuring trunk and vlan ifaces] 2026-03-09 18:22:45.219134 | controller | ok 2026-03-09 18:22:45.224715 | 2026-03-09 18:22:45.224781 | LOOP [Create VLANs ports and attach them to the trunk] 2026-03-09 18:22:45.264777 | controller | skipping: Conditional result was False 2026-03-09 18:22:48.272178 | controller | skipping: Conditional result was False 2026-03-09 18:22:51.292741 | controller | skipping: Conditional result was False 2026-03-09 18:22:51.317907 | 2026-03-09 18:22:51.318182 | LOOP [Prepare VLANs interface data to be used configuring] 2026-03-09 18:22:51.468629 | controller | ok: 2026-03-09 18:22:51.468919 | controller | { 2026-03-09 18:22:51.468961 | controller | "key": "internal-api", 2026-03-09 18:22:51.468989 | controller | "value": { 2026-03-09 18:22:51.469039 | controller | "ip": "172.17.0.5" 2026-03-09 18:22:51.469074 | controller | } 2026-03-09 18:22:51.469102 | controller | } 2026-03-09 18:22:51.469148 | 2026-03-09 18:22:51.496006 | controller | ok: 2026-03-09 18:22:51.496222 | controller | { 2026-03-09 18:22:51.496244 | controller | "key": "storage", 2026-03-09 18:22:51.496262 | controller | "value": { 2026-03-09 18:22:51.496278 | controller | "ip": "172.18.0.5" 2026-03-09 18:22:51.496294 | controller | } 2026-03-09 18:22:51.496310 | controller | } 2026-03-09 18:22:51.527761 | controller | ok: 2026-03-09 18:22:51.527866 | controller | { 2026-03-09 18:22:51.527888 | controller | "key": "tenant", 2026-03-09 18:22:51.527906 | controller | "value": { 2026-03-09 18:22:51.527923 | controller | "ip": "172.19.0.5" 2026-03-09 18:22:51.527939 | controller | } 2026-03-09 18:22:51.527955 | controller | } 2026-03-09 18:22:51.549806 | 2026-03-09 18:22:51.549969 | LOOP [Create NetworkManager configuration file for the trunk port] 2026-03-09 18:22:52.564595 | controller -> 38.102.83.74 | changed: 2026-03-09 18:22:52.564939 | controller -> 38.102.83.74 | { 2026-03-09 18:22:52.564992 | controller -> 38.102.83.74 | "key": "default", 2026-03-09 18:22:52.565070 | controller -> 38.102.83.74 | "value": { 2026-03-09 18:22:52.565109 | controller -> 38.102.83.74 | "connection": "ci-private-network", 2026-03-09 18:22:52.565144 | controller -> 38.102.83.74 | "gw": "192.168.122.1", 2026-03-09 18:22:52.565179 | controller -> 38.102.83.74 | "iface": "ens7", 2026-03-09 18:22:52.565213 | controller -> 38.102.83.74 | "ip": "192.168.122.10/24", 2026-03-09 18:22:52.565245 | controller -> 38.102.83.74 | "mac": "fa:16:3e:d3:67:c2", 2026-03-09 18:22:52.565278 | controller -> 38.102.83.74 | "mtu": "1500" 2026-03-09 18:22:52.565310 | controller -> 38.102.83.74 | } 2026-03-09 18:22:52.565343 | controller -> 38.102.83.74 | } 2026-03-09 18:22:55.661785 | controller -> 38.102.83.74 | changed: 2026-03-09 18:22:55.661905 | controller -> 38.102.83.74 | { 2026-03-09 18:22:55.661928 | controller -> 38.102.83.74 | "key": "internal-api", 2026-03-09 18:22:55.661945 | controller -> 38.102.83.74 | "value": { 2026-03-09 18:22:55.661960 | controller -> 38.102.83.74 | "connection": "ci-private-network-20", 2026-03-09 18:22:55.661976 | controller -> 38.102.83.74 | "iface": "ens7.20", 2026-03-09 18:22:55.661990 | controller -> 38.102.83.74 | "ip": "172.17.0.5/24", 2026-03-09 18:22:55.662005 | controller -> 38.102.83.74 | "mac": "52:54:00:e6:c4:6d", 2026-03-09 18:22:55.662042 | controller -> 38.102.83.74 | "mtu": "1496", 2026-03-09 18:22:55.662058 | controller -> 38.102.83.74 | "parent_iface": "ens7", 2026-03-09 18:22:55.662073 | controller -> 38.102.83.74 | "vlan": 20 2026-03-09 18:22:55.662088 | controller -> 38.102.83.74 | } 2026-03-09 18:22:55.662102 | controller -> 38.102.83.74 | } 2026-03-09 18:22:56.608998 | controller -> 38.102.83.74 | changed: 2026-03-09 18:22:56.609197 | controller -> 38.102.83.74 | { 2026-03-09 18:22:56.609248 | controller -> 38.102.83.74 | "key": "storage", 2026-03-09 18:22:56.609288 | controller -> 38.102.83.74 | "value": { 2026-03-09 18:22:56.609341 | controller -> 38.102.83.74 | "connection": "ci-private-network-21", 2026-03-09 18:22:56.609401 | controller -> 38.102.83.74 | "iface": "ens7.21", 2026-03-09 18:22:56.609438 | controller -> 38.102.83.74 | "ip": "172.18.0.5/24", 2026-03-09 18:22:56.609472 | controller -> 38.102.83.74 | "mac": "52:54:00:af:51:b8", 2026-03-09 18:22:56.609505 | controller -> 38.102.83.74 | "mtu": "1496", 2026-03-09 18:22:56.609537 | controller -> 38.102.83.74 | "parent_iface": "ens7", 2026-03-09 18:22:56.609569 | controller -> 38.102.83.74 | "vlan": 21 2026-03-09 18:22:56.609601 | controller -> 38.102.83.74 | } 2026-03-09 18:22:56.609632 | controller -> 38.102.83.74 | } 2026-03-09 18:22:57.563309 | controller -> 38.102.83.74 | changed: 2026-03-09 18:22:57.563435 | controller -> 38.102.83.74 | { 2026-03-09 18:22:57.563476 | controller -> 38.102.83.74 | "key": "tenant", 2026-03-09 18:22:57.563504 | controller -> 38.102.83.74 | "value": { 2026-03-09 18:22:57.563524 | controller -> 38.102.83.74 | "connection": "ci-private-network-22", 2026-03-09 18:22:57.563542 | controller -> 38.102.83.74 | "iface": "ens7.22", 2026-03-09 18:22:57.563559 | controller -> 38.102.83.74 | "ip": "172.19.0.5/24", 2026-03-09 18:22:57.563575 | controller -> 38.102.83.74 | "mac": "52:54:00:59:a3:dc", 2026-03-09 18:22:57.563592 | controller -> 38.102.83.74 | "mtu": "1496", 2026-03-09 18:22:57.563607 | controller -> 38.102.83.74 | "parent_iface": "ens7", 2026-03-09 18:22:57.563621 | controller -> 38.102.83.74 | "vlan": 22 2026-03-09 18:22:57.563637 | controller -> 38.102.83.74 | } 2026-03-09 18:22:57.563652 | controller -> 38.102.83.74 | } 2026-03-09 18:22:57.576249 | 2026-03-09 18:22:57.576409 | TASK [Refresh NetworkManager] 2026-03-09 18:22:58.366394 | controller -> 38.102.83.74 | changed 2026-03-09 18:22:58.383535 | 2026-03-09 18:22:58.383679 | TASK [Debug fetch IP routes] 2026-03-09 18:22:58.718552 | controller -> 38.102.83.74 | changed 2026-03-09 18:22:58.731770 | 2026-03-09 18:22:58.731944 | TASK [Debug IP routes] 2026-03-09 18:22:58.769123 | controller -> 38.102.83.74 | ok 2026-03-09 18:22:58.795587 | 2026-03-09 18:22:58.795726 | TASK [Display some data about network ports] 2026-03-09 18:23:00.647022 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+ 2026-03-09 18:23:00.751429 | controller | | ID | Name | MAC Address | Fixed IP Addresses | Status | 2026-03-09 18:23:00.751563 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+ 2026-03-09 18:23:00.751571 | controller | | 1819571a-0111-4e66-a077-2cec5c260d6d | | fa:16:3e:f5:a2:32 | | DOWN | 2026-03-09 18:23:00.751577 | controller | | 2a0fed40-01ed-4d11-863e-67f7d40991d7 | crc-ea3e2df3-251a-4cdb-9064-3c52ac509aba | fa:16:3e:d3:67:c2 | ip_address='192.168.122.10', subnet_id='edcfb140-8934-4cc7-a46a-d8cf20ecfdf7' | ACTIVE | 2026-03-09 18:23:00.751582 | controller | | 6d648f67-f72a-4d32-8184-c2d17ffb3d8c | controller-593e5d4e-760b-40a9-aa62-62e7bf2adfde | fa:16:3e:08:be:f9 | ip_address='192.168.122.11', subnet_id='edcfb140-8934-4cc7-a46a-d8cf20ecfdf7' | ACTIVE | 2026-03-09 18:23:00.751586 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+ 2026-03-09 18:23:00.841675 | controller | changed 2026-03-09 18:23:00.850940 | 2026-03-09 18:23:00.851122 | LOOP [Display server configuration] 2026-03-09 18:23:03.346468 | controller | changed 2026-03-09 18:23:03.346710 | controller | changed: All items complete 2026-03-09 18:23:03.346763 | 2026-03-09 18:23:08.426522 | controller | changed 2026-03-09 18:23:08.439597 | 2026-03-09 18:23:08.439718 | TASK [Fetch underneath provider DNSs] 2026-03-09 18:23:08.460291 | controller | ok 2026-03-09 18:23:08.468443 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-fetch-provider-dns.yml 2026-03-09 18:23:08.479278 | 2026-03-09 18:23:08.479351 | TASK [Fetch the DNS list of the default interface] 2026-03-09 18:23:08.873822 | controller | ok 2026-03-09 18:23:08.888259 | 2026-03-09 18:23:08.888676 | TASK [Set provider DNS for further usage] 2026-03-09 18:23:08.943561 | controller | ok 2026-03-09 18:23:08.949157 | 2026-03-09 18:23:08.949246 | LOOP [Save networking data to file for further usage] 2026-03-09 18:23:09.814763 | controller -> 38.102.83.74 | changed: "crc" 2026-03-09 18:23:10.466900 | controller -> 38.102.83.2 | changed: "controller" 2026-03-09 18:23:10.489462 | 2026-03-09 18:23:10.489626 | TASK [Remove cloud_secrets file] 2026-03-09 18:23:10.748704 | controller | changed 2026-03-09 18:23:10.784270 | 2026-03-09 18:23:10.784393 | PLAY RECAP 2026-03-09 18:23:10.784452 | controller | ok: 48 changed: 22 unreachable: 0 failed: 0 skipped: 14 rescued: 0 ignored: 0 2026-03-09 18:23:10.784480 | 2026-03-09 18:23:10.959526 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network.yml@master] 2026-03-09 18:23:10.964962 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/prepare-crc.yaml@master] 2026-03-09 18:23:11.572198 | 2026-03-09 18:23:11.572370 | PLAY [Prefer ipv4 over ipv6] 2026-03-09 18:23:11.623367 | 2026-03-09 18:23:11.623508 | TASK [Configure /etc/gai.conf to prever ipv4 over ipv6] 2026-03-09 18:23:11.680245 | controller | skipping: Conditional result was False 2026-03-09 18:23:11.701283 | crc | skipping: Conditional result was False 2026-03-09 18:23:11.762712 | 2026-03-09 18:23:11.762837 | PLAY [Generic task for migration time to crc-cloud] 2026-03-09 18:23:11.784635 | 2026-03-09 18:23:11.784767 | TASK [Check if there is a image build date file] 2026-03-09 18:23:12.327404 | crc | ok 2026-03-09 18:23:12.342114 | 2026-03-09 18:23:12.342302 | TASK [Print image date] 2026-03-09 18:23:12.808293 | crc | Mon Mar 9 01:03:15 UTC 2026 | Image type: crc-cloud 2026-03-09 18:23:12.921512 | crc | ok 2026-03-09 18:23:12.929324 | 2026-03-09 18:23:12.929395 | TASK [Run crc-cloud preparation] 2026-03-09 18:23:12.972352 | crc | ok 2026-03-09 18:23:13.026103 | 2026-03-09 18:23:13.026233 | TASK [prepare-crc-cloud : Print build date] 2026-03-09 18:23:13.051227 | crc | ok 2026-03-09 18:23:13.062844 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/build_date.yaml 2026-03-09 18:23:13.076516 | 2026-03-09 18:23:13.076647 | TASK [prepare-crc-cloud : Check if there is a image build date file] 2026-03-09 18:23:13.410995 | crc | ok 2026-03-09 18:23:13.429400 | 2026-03-09 18:23:13.429554 | TASK [prepare-crc-cloud : Print image date if available] 2026-03-09 18:23:13.763735 | crc | Mon Mar 9 01:03:15 UTC 2026 | Image type: crc-cloud 2026-03-09 18:23:14.002811 | crc | ok 2026-03-09 18:23:14.020342 | 2026-03-09 18:23:14.020529 | TASK [prepare-crc-cloud : Tune Kubelet config file] 2026-03-09 18:23:14.058126 | crc | skipping: Conditional result was False 2026-03-09 18:23:14.076276 | 2026-03-09 18:23:14.076517 | TASK [prepare-crc-cloud : Reboot CRC host to get the network interfaces] 2026-03-09 18:23:14.114238 | crc | ok 2026-03-09 18:23:14.127920 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/reboot_crc.yaml 2026-03-09 18:23:14.149074 | 2026-03-09 18:23:14.149218 | TASK [prepare-crc-cloud : Reboot host after adding pull secret and changing kubelet node params] 2026-03-09 18:23:50.200588 | crc | changed 2026-03-09 18:23:50.219119 | 2026-03-09 18:23:50.219302 | TASK [Start Zuul console after reboot] 2026-03-09 18:23:50.265398 | crc | ok 2026-03-09 18:23:50.301079 | 2026-03-09 18:23:50.301220 | TASK [start-zuul-console : Start zuul_console daemon.] 2026-03-09 18:23:50.917539 | crc | ok 2026-03-09 18:23:50.949573 | 2026-03-09 18:23:50.949702 | TASK [prepare-crc-cloud : Prepare for executing crc-cloud] 2026-03-09 18:23:50.991825 | crc | ok 2026-03-09 18:23:51.007154 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/pre.yaml 2026-03-09 18:23:51.024827 | 2026-03-09 18:23:51.024978 | TASK [prepare-crc-cloud : Disable overwriting /etc/resolv.conf by the NetworkManager] 2026-03-09 18:23:52.123716 | crc | changed 2026-03-09 18:23:52.131332 | 2026-03-09 18:23:52.131398 | TASK [prepare-crc-cloud : Reload NetworkManager after creating custom rules] 2026-03-09 18:23:53.042538 | crc | changed 2026-03-09 18:23:53.057732 | 2026-03-09 18:23:53.057882 | TASK [prepare-crc-cloud : Inject pull secret into /var/lib/kubelet/config.json] 2026-03-09 18:23:54.741577 | crc | changed 2026-03-09 18:23:54.760918 | 2026-03-09 18:23:54.761099 | TASK [prepare-crc-cloud : Check if etcd slow profile set] 2026-03-09 18:23:54.797305 | crc | skipping: Conditional result was False 2026-03-09 18:23:54.817454 | 2026-03-09 18:23:54.817664 | TASK [prepare-crc-cloud : Use ramdisk for etcd] 2026-03-09 18:23:54.849133 | crc | ok 2026-03-09 18:23:54.862036 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/ramdisk.yaml 2026-03-09 18:23:54.915979 | 2026-03-09 18:23:54.916122 | TASK [prepare-crc-cloud : Move current etcd to other place] 2026-03-09 18:23:55.503273 | crc | ok 2026-03-09 18:23:55.520225 | 2026-03-09 18:23:55.520392 | TASK [prepare-crc-cloud : Mount ramdisk] 2026-03-09 18:23:55.956353 | crc | changed 2026-03-09 18:23:55.974609 | 2026-03-09 18:23:55.974800 | TASK [prepare-crc-cloud : Set proper permissions after mount] 2026-03-09 18:23:56.434359 | crc | changed 2026-03-09 18:23:56.451581 | 2026-03-09 18:23:56.451746 | TASK [prepare-crc-cloud : Copy content from old location to ramdisk] 2026-03-09 18:23:57.986706 | crc | changed 2026-03-09 18:23:57.995525 | 2026-03-09 18:23:57.995633 | TASK [prepare-crc-cloud : Set proper SELinux context] 2026-03-09 18:23:58.583474 | crc | ok 2026-03-09 18:23:58.593248 | 2026-03-09 18:23:58.593364 | TASK [prepare-crc-cloud : Block crc.testing domains] 2026-03-09 18:23:58.628270 | crc | ok 2026-03-09 18:23:58.639894 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/block_crc_domains.yaml 2026-03-09 18:23:58.658407 | 2026-03-09 18:23:58.658524 | LOOP [prepare-crc-cloud : Set crc.testing domains into /etc/hosts] 2026-03-09 18:23:59.161666 | crc | changed: line added 2026-03-09 18:23:59.456747 | crc | changed: line added 2026-03-09 18:23:59.712609 | crc | changed: line added 2026-03-09 18:24:00.021269 | crc | changed: line added 2026-03-09 18:24:00.330997 | crc | changed: line added 2026-03-09 18:24:00.618913 | crc | changed: line added 2026-03-09 18:24:00.905001 | crc | changed: line added 2026-03-09 18:24:01.195895 | crc | changed: line added 2026-03-09 18:24:01.208610 | 2026-03-09 18:24:01.208721 | TASK [prepare-crc-cloud : Get default interface name] 2026-03-09 18:24:01.676851 | crc | ens3 2026-03-09 18:24:01.831724 | crc | ok 2026-03-09 18:24:01.840394 | 2026-03-09 18:24:01.840525 | LOOP [prepare-crc-cloud : Block CRC domains also using iptables] 2026-03-09 18:24:02.383592 | crc | ok 2026-03-09 18:24:02.383825 | crc | changed: All items complete 2026-03-09 18:24:02.383862 | 2026-03-09 18:24:02.671270 | crc | ok 2026-03-09 18:24:02.963934 | crc | ok 2026-03-09 18:24:03.303562 | crc | ok 2026-03-09 18:24:03.612997 | crc | ok 2026-03-09 18:24:03.907351 | crc | ok 2026-03-09 18:24:04.199213 | crc | ok 2026-03-09 18:24:04.486194 | crc | ok 2026-03-09 18:24:04.504403 | 2026-03-09 18:24:04.504588 | TASK [prepare-crc-cloud : Set required facts] 2026-03-09 18:24:04.533090 | crc | ok 2026-03-09 18:24:04.546323 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/set_vars.yaml 2026-03-09 18:24:04.576113 | 2026-03-09 18:24:04.576282 | TASK [prepare-crc-cloud : Set required facts] 2026-03-09 18:24:04.631479 | crc | Output suppressed because no_log was given 2026-03-09 18:24:04.641645 | 2026-03-09 18:24:04.641779 | TASK [Start crc-cloud role when Zuul defined] 2026-03-09 18:24:04.687958 | crc | ok 2026-03-09 18:24:04.755899 | 2026-03-09 18:24:04.756094 | TASK [deploy-crc-cloud : Check if pull-secret is provided] 2026-03-09 18:24:04.781949 | crc | skipping: Conditional result was False 2026-03-09 18:24:04.793857 | 2026-03-09 18:24:04.794019 | TASK [deploy-crc-cloud : Create kubeconfig] 2026-03-09 18:24:04.826584 | crc | ok 2026-03-09 18:24:04.836552 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/kubeconfig.yaml 2026-03-09 18:24:04.849228 | 2026-03-09 18:24:04.849325 | TASK [deploy-crc-cloud : Create kube directory] 2026-03-09 18:24:05.134354 | crc | changed 2026-03-09 18:24:05.142128 | 2026-03-09 18:24:05.142234 | TASK [deploy-crc-cloud : Copy kubeconfig to user dir] 2026-03-09 18:24:05.573625 | crc | changed 2026-03-09 18:24:05.583622 | 2026-03-09 18:24:05.583739 | TASK [deploy-crc-cloud : Setup dnsmasq] 2026-03-09 18:24:05.618316 | crc | ok 2026-03-09 18:24:05.636107 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/dnsmasq.yaml 2026-03-09 18:24:05.655679 | 2026-03-09 18:24:05.655792 | TASK [deploy-crc-cloud : Disable overwriting /etc/resolv.conf by the NetworkManager] 2026-03-09 18:24:06.446238 | crc | ok 2026-03-09 18:24:06.467273 | 2026-03-09 18:24:06.467524 | TASK [deploy-crc-cloud : Restart NetworkManager when its needed] 2026-03-09 18:24:06.494655 | crc | skipping: Conditional result was False 2026-03-09 18:24:06.507765 | 2026-03-09 18:24:06.507936 | TASK [deploy-crc-cloud : Ensure /etc/resolv.conf is not a symlink and is writable] 2026-03-09 18:24:06.818064 | crc | ok 2026-03-09 18:24:06.879210 | 2026-03-09 18:24:06.879360 | TASK [deploy-crc-cloud : Create crc-dnsmasq.conf] 2026-03-09 18:24:07.779087 | crc | changed 2026-03-09 18:24:07.793731 | 2026-03-09 18:24:07.793901 | TASK [deploy-crc-cloud : Add domains into crc-dnsmasq.conf when alternative_domain] 2026-03-09 18:24:07.820338 | crc | skipping: Conditional result was False 2026-03-09 18:24:07.834019 | 2026-03-09 18:24:07.834232 | LOOP [deploy-crc-cloud : Set this host as first nameserver in /etc/resolv.conf] 2026-03-09 18:24:08.186148 | crc | changed: line replaced 2026-03-09 18:24:08.441907 | crc | ok 2026-03-09 18:24:08.752650 | crc | ok 2026-03-09 18:24:08.773969 | 2026-03-09 18:24:08.774129 | TASK [deploy-crc-cloud : Restart dnsmasq] 2026-03-09 18:24:09.647936 | crc | changed 2026-03-09 18:24:09.661605 | 2026-03-09 18:24:09.661741 | TASK [deploy-crc-cloud : Start kubelet] 2026-03-09 18:24:09.696848 | crc | ok 2026-03-09 18:24:09.710323 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/kubelet.yaml 2026-03-09 18:24:09.735267 | 2026-03-09 18:24:09.735394 | TASK [deploy-crc-cloud : Start and enable kubelet] 2026-03-09 18:24:23.555871 | crc | changed 2026-03-09 18:24:23.577129 | 2026-03-09 18:24:23.577272 | TASK [deploy-crc-cloud : Wait for port 6443 to be up] 2026-03-09 18:25:24.143738 | crc | ok 2026-03-09 18:25:24.160526 | 2026-03-09 18:25:24.160703 | TASK [deploy-crc-cloud : Wait for API to start before continue] 2026-03-09 18:25:25.549241 | crc | NAMESPACE NAME READY STATUS RESTARTS AGE 2026-03-09 18:25:25.549312 | crc | openshift-apiserver-operator openshift-apiserver-operator-796bbdcf4f-vlwv5 0/1 Pending 0 5s 2026-03-09 18:25:25.549363 | crc | openshift-apiserver apiserver-76f77b778f-m6q6r 0/2 Pending 0 5s 2026-03-09 18:25:25.549382 | crc | openshift-authentication-operator authentication-operator-69f744f599-jrr9g 0/1 Pending 0 5s 2026-03-09 18:25:25.549394 | crc | openshift-authentication oauth-openshift-558db77b4-tnl4x 0/1 Pending 0 5s 2026-03-09 18:25:25.549405 | crc | openshift-cluster-machine-approver machine-approver-56656f9798-nprxk 0/2 Pending 0 5s 2026-03-09 18:25:25.549415 | crc | openshift-cluster-samples-operator cluster-samples-operator-665b6dd947-gbjt5 0/2 Pending 0 5s 2026-03-09 18:25:25.549426 | crc | openshift-cluster-version cluster-version-operator-5c965bbfc6-qnxgv 0/1 Pending 0 5s 2026-03-09 18:25:25.549439 | crc | openshift-config-operator openshift-config-operator-7777fb866f-l97hl 0/1 Pending 0 5s 2026-03-09 18:25:25.549448 | crc | openshift-console-operator console-operator-58897d9998-gg4ds 0/1 Pending 0 5s 2026-03-09 18:25:25.549457 | crc | openshift-console console-f9d7485db-x9nnw 0/1 Pending 0 5s 2026-03-09 18:25:25.549474 | crc | openshift-console downloads-7954f5f757-295wb 0/1 Pending 0 5s 2026-03-09 18:25:25.549486 | crc | openshift-controller-manager-operator openshift-controller-manager-operator-756b6f6bc6-9pcrm 0/1 Pending 0 5s 2026-03-09 18:25:25.549510 | crc | openshift-controller-manager controller-manager-879f6c89f-vmzvr 0/1 Pending 0 5s 2026-03-09 18:25:25.549594 | crc | openshift-dns-operator dns-operator-744455d44c-znqzp 0/2 Pending 0 5s 2026-03-09 18:25:25.549612 | crc | openshift-dns node-resolver-n9tvt 0/1 Pending 0 5s 2026-03-09 18:25:25.549623 | crc | openshift-etcd-operator etcd-operator-b45778765-sh5rp 0/1 Pending 0 5s 2026-03-09 18:25:25.549637 | crc | openshift-image-registry cluster-image-registry-operator-dc59b4c8b-66c4m 0/1 Pending 0 5s 2026-03-09 18:25:25.549651 | crc | openshift-image-registry image-registry-697d97f7c8-xbxp5 0/1 Pending 0 5s 2026-03-09 18:25:25.549695 | crc | openshift-image-registry node-ca-mfdmq 0/1 Pending 0 5s 2026-03-09 18:25:25.549706 | crc | openshift-ingress-operator ingress-operator-5b745b69d9-7fdbf 0/2 Pending 0 5s 2026-03-09 18:25:25.549721 | crc | openshift-ingress router-default-5444994796-4ntmx 0/1 Pending 0 5s 2026-03-09 18:25:25.549755 | crc | openshift-kube-apiserver-operator kube-apiserver-operator-766d6c64bb-xxsvf 0/1 Pending 0 5s 2026-03-09 18:25:25.549778 | crc | openshift-kube-controller-manager-operator kube-controller-manager-operator-78b949d7b-ctxcr 0/1 Pending 0 5s 2026-03-09 18:25:25.549800 | crc | openshift-kube-scheduler-operator openshift-kube-scheduler-operator-5fdd9b5758-46tk5 0/1 Pending 0 5s 2026-03-09 18:25:25.549823 | crc | openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-b67b599dd-gk67k 0/1 Pending 0 5s 2026-03-09 18:25:25.549852 | crc | openshift-kube-storage-version-migrator migrator-59844c95c7-n5glb 0/2 Pending 0 5s 2026-03-09 18:25:25.549884 | crc | openshift-machine-api control-plane-machine-set-operator-78cbb6b69f-dbk5m 0/1 Pending 0 5s 2026-03-09 18:25:25.549922 | crc | openshift-machine-api machine-api-operator-5694c8668f-h8j2t 0/2 Pending 0 5s 2026-03-09 18:25:25.549951 | crc | openshift-machine-config-operator machine-config-controller-84d6567774-ppt7p 0/2 Pending 0 5s 2026-03-09 18:25:25.549987 | crc | openshift-machine-config-operator machine-config-daemon-kk7gs 0/2 Pending 0 5s 2026-03-09 18:25:25.550017 | crc | openshift-machine-config-operator machine-config-operator-74547568cd-5ncxl 0/2 Pending 0 5s 2026-03-09 18:25:25.550057 | crc | openshift-marketplace marketplace-operator-79b997595-7nw2x 0/1 Pending 0 5s 2026-03-09 18:25:25.550086 | crc | openshift-multus multus-additional-cni-plugins-b9gd4 0/1 Pending 0 5s 2026-03-09 18:25:25.550119 | crc | openshift-multus multus-admission-controller-857f4d67dd-d9kvs 0/2 Pending 0 5s 2026-03-09 18:25:25.550162 | crc | openshift-multus multus-lw2hk 0/1 Pending 0 5s 2026-03-09 18:25:25.550199 | crc | openshift-multus network-metrics-daemon-lf7bd 0/2 Pending 0 5s 2026-03-09 18:25:25.550227 | crc | openshift-network-console networking-console-plugin-85b44fc459-gdk6g 1/1 Running 2 379d 2026-03-09 18:25:25.550261 | crc | openshift-network-diagnostics network-check-source-55646444c4-trplf 1/1 Running 2 379d 2026-03-09 18:25:25.550295 | crc | openshift-network-diagnostics network-check-target-xd92c 1/1 Running 2 379d 2026-03-09 18:25:25.550354 | crc | openshift-network-node-identity network-node-identity-vrzqb 2/2 Running 7 379d 2026-03-09 18:25:25.550393 | crc | openshift-network-operator iptables-alerter-4ln5h 1/1 Running 2 379d 2026-03-09 18:25:25.550426 | crc | openshift-network-operator network-operator-58b4c7f79c-55gtf 1/1 Running 4 379d 2026-03-09 18:25:25.550463 | crc | openshift-oauth-apiserver apiserver-7bbb656c7d-bsmz7 0/1 Pending 0 5s 2026-03-09 18:25:25.550491 | crc | openshift-operator-lifecycle-manager catalog-operator-68c6474976-jrc42 0/1 Pending 0 5s 2026-03-09 18:25:25.550523 | crc | openshift-operator-lifecycle-manager collect-profiles-29551335-b9jvf 0/1 Pending 0 5s 2026-03-09 18:25:25.550557 | crc | openshift-operator-lifecycle-manager olm-operator-6b444d44fb-x76nr 0/1 Pending 0 5s 2026-03-09 18:25:25.550591 | crc | openshift-operator-lifecycle-manager package-server-manager-789f6589d5-ft9v6 0/2 Pending 0 5s 2026-03-09 18:25:25.550624 | crc | openshift-operator-lifecycle-manager packageserver-d55dfcdfc-2lkq2 0/1 Pending 0 5s 2026-03-09 18:25:25.550659 | crc | openshift-ovn-kubernetes ovnkube-control-plane-749d76644c-d6g54 0/2 Pending 0 5s 2026-03-09 18:25:25.550693 | crc | openshift-ovn-kubernetes ovnkube-node-bfdsp 0/8 Pending 0 5s 2026-03-09 18:25:25.550726 | crc | openshift-route-controller-manager route-controller-manager-6576b87f9c-qqsgs 0/1 Pending 0 5s 2026-03-09 18:25:25.550758 | crc | openshift-service-ca-operator service-ca-operator-777779d784-hh9sf 0/1 Pending 0 5s 2026-03-09 18:25:25.550823 | crc | openshift-service-ca service-ca-9c57cc56f-4pwqq 0/1 Pending 0 5s 2026-03-09 18:25:25.710584 | crc | ok 2026-03-09 18:25:25.719148 | 2026-03-09 18:25:25.719246 | TASK [deploy-crc-cloud : Replace default pubkey] 2026-03-09 18:25:25.744998 | crc | ok 2026-03-09 18:25:25.755837 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/pubkey.yaml 2026-03-09 18:25:25.775831 | 2026-03-09 18:25:25.775982 | TASK [deploy-crc-cloud : Check if id_rsa.pub exists] 2026-03-09 18:25:26.005263 | crc | ok 2026-03-09 18:25:26.017540 | 2026-03-09 18:25:26.017772 | TASK [deploy-crc-cloud : Read pub key] 2026-03-09 18:25:26.578135 | crc | skipping: Conditional result was False 2026-03-09 18:25:26.595056 | 2026-03-09 18:25:26.595242 | TASK [deploy-crc-cloud : Wait for machineconfig] 2026-03-09 18:25:26.632661 | crc | skipping: Conditional result was False 2026-03-09 18:25:26.646345 | 2026-03-09 18:25:26.646465 | TASK [deploy-crc-cloud : Patch machineconfig 99-master-ssh] 2026-03-09 18:25:27.234834 | crc | skipping: Conditional result was False 2026-03-09 18:25:27.243128 | 2026-03-09 18:25:27.243203 | TASK [deploy-crc-cloud : Accept certificate] 2026-03-09 18:25:27.343876 | crc | ok 2026-03-09 18:25:27.359684 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/accept_cert.yaml 2026-03-09 18:25:27.392039 | 2026-03-09 18:25:27.392181 | LOOP [deploy-crc-cloud : Re-run accept certificates in Pending state few times] 2026-03-09 18:25:28.581828 | crc | ok 2026-03-09 18:25:28.582116 | crc | changed: All items complete 2026-03-09 18:25:28.582148 | 2026-03-09 18:25:39.189582 | crc | ok 2026-03-09 18:25:49.634415 | crc | ok 2026-03-09 18:25:49.656387 | 2026-03-09 18:25:49.656554 | TASK [deploy-crc-cloud : Create service account for - auto-csr-approver] 2026-03-09 18:25:50.529185 | crc | changed 2026-03-09 18:25:50.546904 | 2026-03-09 18:25:50.547123 | TASK [deploy-crc-cloud : Apply cluster cert approver] 2026-03-09 18:25:51.090558 | crc | serviceaccount/csr-approver-sa created 2026-03-09 18:25:51.098744 | crc | clusterrole.rbac.authorization.k8s.io/csr-approver created 2026-03-09 18:25:51.106852 | crc | clusterrolebinding.rbac.authorization.k8s.io/csr-approver-binding created 2026-03-09 18:25:51.120830 | crc | cronjob.batch/auto-csr-approver created 2026-03-09 18:25:51.609132 | crc | ok 2026-03-09 18:25:51.620391 | 2026-03-09 18:25:51.620660 | TASK [deploy-crc-cloud : Wait for cluster become healthy] 2026-03-09 18:25:51.648598 | crc | ok 2026-03-09 18:25:51.663558 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml 2026-03-09 18:25:51.697633 | 2026-03-09 18:25:51.697754 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver] 2026-03-09 18:25:52.263865 | crc | True 2026-03-09 18:25:52.264808 | crc | False 2026-03-09 18:27:25.167363 | crc | changed 2026-03-09 18:27:25.174780 | 2026-03-09 18:27:25.174879 | TASK [deploy-crc-cloud : Output success message if components are healthy] 2026-03-09 18:27:25.243185 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds. 2026-03-09 18:27:25.256836 | 2026-03-09 18:27:25.256954 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy] 2026-03-09 18:27:25.316910 | crc | skipping: Conditional result was False 2026-03-09 18:27:25.325438 | 2026-03-09 18:27:25.325547 | TASK [deploy-crc-cloud : Set credentials] 2026-03-09 18:27:25.379589 | crc | ok 2026-03-09 18:27:25.406624 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/set_credentials.yaml 2026-03-09 18:27:25.458520 | 2026-03-09 18:27:25.458637 | TASK [deploy-crc-cloud : Ensure htpasswd.txt does not exists] 2026-03-09 18:27:25.789137 | crc | ok 2026-03-09 18:27:25.799156 | 2026-03-09 18:27:25.799261 | TASK [deploy-crc-cloud : Create temporary directory] 2026-03-09 18:27:26.295090 | crc | changed 2026-03-09 18:27:26.302802 | 2026-03-09 18:27:26.302889 | LOOP [deploy-crc-cloud : Get htpasswd] 2026-03-09 18:27:26.392793 | crc | Output suppressed because no_log was given 2026-03-09 18:27:26.411301 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml 2026-03-09 18:27:26.412336 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml 2026-03-09 18:27:26.413281 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml 2026-03-09 18:27:26.434856 | 2026-03-09 18:27:26.434949 | TASK [deploy-crc-cloud : Get htpasswd for developer] 2026-03-09 18:27:37.941545 | crc | time="2026-03-09T18:27:37Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-03-09 18:27:37.941776 | crc | Trying to pull ghcr.io/crc-org/crc-cloud/httpd-tools:latest... 2026-03-09 18:27:38.494215 | crc | Getting image source signatures 2026-03-09 18:27:38.494623 | crc | Copying blob sha256:1730cf1a00b8c06596486f3456376e7660a42976bb9fcd6604ae22a70fca6b65 2026-03-09 18:27:38.494715 | crc | Copying blob sha256:f391d299bb65afd981883c104f434a04ee3ffb247e341247171779ef526ed65f 2026-03-09 18:27:40.789869 | crc | Copying config sha256:d81064f8940a06af9e1a39fc510c6bfc299bb166f99ee6a892f22ec84ee58041 2026-03-09 18:27:40.800053 | crc | Writing manifest to image destination 2026-03-09 18:27:41.528169 | crc | changed 2026-03-09 18:27:41.547575 | 2026-03-09 18:27:41.547757 | TASK [deploy-crc-cloud : Get htpasswd for kubeadmin] 2026-03-09 18:27:42.093490 | crc | time="2026-03-09T18:27:42Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-03-09 18:27:42.711329 | crc | changed 2026-03-09 18:27:42.722708 | 2026-03-09 18:27:42.722843 | TASK [deploy-crc-cloud : Get htpasswd for redhat] 2026-03-09 18:27:43.191473 | crc | time="2026-03-09T18:27:43Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-03-09 18:27:43.789903 | crc | changed 2026-03-09 18:27:43.823380 | 2026-03-09 18:27:43.823612 | TASK [deploy-crc-cloud : Cleanup htpasswd.txt file] 2026-03-09 18:27:44.409151 | crc | changed 2026-03-09 18:27:44.428814 | 2026-03-09 18:27:44.429342 | TASK [deploy-crc-cloud : Create secret with generic htpass-secret] 2026-03-09 18:27:45.509792 | crc | ok 2026-03-09 18:27:45.522668 | 2026-03-09 18:27:45.522822 | TASK [deploy-crc-cloud : Replace htpass-secret] 2026-03-09 18:27:45.983082 | crc | secret/htpass-secret replaced 2026-03-09 18:27:46.097609 | crc | ok 2026-03-09 18:27:46.106211 | 2026-03-09 18:27:46.106304 | TASK [deploy-crc-cloud : Remove temporary directory] 2026-03-09 18:27:46.464704 | crc | changed 2026-03-09 18:27:46.478795 | 2026-03-09 18:27:46.479065 | TASK [deploy-crc-cloud : Replace default CA] 2026-03-09 18:27:46.513788 | crc | ok 2026-03-09 18:27:46.539498 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/replace_default_ca.yaml 2026-03-09 18:27:46.580321 | 2026-03-09 18:27:46.580470 | TASK [deploy-crc-cloud : Generate key] 2026-03-09 18:27:48.182639 | crc | changed 2026-03-09 18:27:48.218772 | 2026-03-09 18:27:48.218940 | TASK [deploy-crc-cloud : Generate cert] 2026-03-09 18:27:48.785670 | crc | changed 2026-03-09 18:27:48.793797 | 2026-03-09 18:27:48.793876 | TASK [deploy-crc-cloud : Generate csr] 2026-03-09 18:27:49.189549 | crc | ..................+..+...+.......+..+.+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.......+...+...+..+......+.+...+...........+................+...+.....+.......+......+.....+..........+............+..+....+.....+....+.....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2026-03-09 18:27:49.285169 | crc | ........+...........+......+...+..........+.....+...+...+....+.....+.+.........+..+....+......+......+..+.............+.....+....+..............+..........+..+.........+.........+.........+.......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+..+...+................+...+...+..+......+......+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+...+.........+...+......+....+..+.+.....+...+............+............+...+.......+...+...+..+....+......+.................+..........+..+.+..+..........+..+.+........+.........+.+..+...................+...........+....+.....+..........+..............+...................+..+....+.....+..........+..+............+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2026-03-09 18:27:49.288336 | crc | ----- 2026-03-09 18:27:49.339103 | crc | changed 2026-03-09 18:27:49.349300 | 2026-03-09 18:27:49.349390 | TASK [deploy-crc-cloud : Generate user cert] 2026-03-09 18:27:49.683116 | crc | Certificate request self-signature ok 2026-03-09 18:27:49.692613 | crc | subject=O = ${GROUP}, CN = core 2026-03-09 18:27:49.913495 | crc | changed 2026-03-09 18:27:49.921597 | 2026-03-09 18:27:49.921674 | TASK [deploy-crc-cloud : Create configmap] 2026-03-09 18:27:50.407998 | crc | configmap/client-ca-custom created 2026-03-09 18:27:50.491503 | crc | ok 2026-03-09 18:27:50.504753 | 2026-03-09 18:27:50.504884 | TASK [deploy-crc-cloud : Patch apiserver] 2026-03-09 18:27:50.984143 | crc | apiserver.config.openshift.io/cluster patched 2026-03-09 18:27:51.112753 | crc | ok 2026-03-09 18:27:51.124271 | 2026-03-09 18:27:51.124427 | TASK [deploy-crc-cloud : Create configmap] 2026-03-09 18:27:51.689451 | crc | W0309 18:27:51.689257 13243 helpers.go:703] --dry-run is deprecated and can be replaced with --dry-run=client. 2026-03-09 18:27:51.729935 | crc | configmap/admin-kubeconfig-client-ca replaced 2026-03-09 18:27:51.797473 | crc | ok 2026-03-09 18:27:51.805209 | 2026-03-09 18:27:51.805275 | TASK [deploy-crc-cloud : Login to the OpenShift cluster] 2026-03-09 18:27:51.846451 | crc | ok 2026-03-09 18:27:51.859417 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/login.yaml 2026-03-09 18:27:51.902604 | 2026-03-09 18:27:51.902716 | TASK [deploy-crc-cloud : Try to login after all changes] 2026-03-09 18:27:52.444038 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported! 2026-03-09 18:27:52.699090 | crc | 2026-03-09 18:27:52.699146 | crc | Login failed (401 Unauthorized) 2026-03-09 18:27:52.708979 | crc | Verify you have provided the correct credentials. 2026-03-09 18:29:25.834543 | crc | ok 2026-03-09 18:29:25.850977 | 2026-03-09 18:29:25.851214 | TASK [deploy-crc-cloud : Patch pull secret] 2026-03-09 18:29:25.889907 | crc | ok 2026-03-09 18:29:25.903222 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/patch_pull_secret.yaml 2026-03-09 18:29:25.946427 | 2026-03-09 18:29:25.946551 | TASK [deploy-crc-cloud : Patch pull secret] 2026-03-09 18:29:26.529955 | crc | Output suppressed because no_log was given 2026-03-09 18:29:26.546610 | 2026-03-09 18:29:26.546794 | TASK [deploy-crc-cloud : Wait for cluster become healthy after patching CA and pull secret] 2026-03-09 18:29:26.600747 | crc | ok 2026-03-09 18:29:26.620862 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml 2026-03-09 18:29:26.666846 | 2026-03-09 18:29:26.666974 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver] 2026-03-09 18:29:27.128185 | crc | True 2026-03-09 18:29:27.129112 | crc | True 2026-03-09 18:29:27.205963 | crc | changed 2026-03-09 18:29:27.214447 | 2026-03-09 18:29:27.214540 | TASK [deploy-crc-cloud : Output success message if components are healthy] 2026-03-09 18:29:27.254788 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds. 2026-03-09 18:29:27.272196 | 2026-03-09 18:29:27.272348 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy] 2026-03-09 18:29:27.297927 | crc | skipping: Conditional result was False 2026-03-09 18:29:27.311342 | 2026-03-09 18:29:27.311500 | TASK [deploy-crc-cloud : Create certificate and patch secret] 2026-03-09 18:29:27.336772 | crc | skipping: Conditional result was False 2026-03-09 18:29:27.348963 | 2026-03-09 18:29:27.349167 | TASK [deploy-crc-cloud : Wait for cluster become healthy after adding domain] 2026-03-09 18:29:27.376726 | crc | ok 2026-03-09 18:29:27.388888 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml 2026-03-09 18:29:27.429973 | 2026-03-09 18:29:27.430127 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver] 2026-03-09 18:29:27.883119 | crc | True 2026-03-09 18:29:27.884260 | crc | True 2026-03-09 18:29:27.970530 | crc | changed 2026-03-09 18:29:27.981301 | 2026-03-09 18:29:27.981417 | TASK [deploy-crc-cloud : Output success message if components are healthy] 2026-03-09 18:29:28.040133 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds. 2026-03-09 18:29:28.057430 | 2026-03-09 18:29:28.057637 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy] 2026-03-09 18:29:28.074354 | crc | skipping: Conditional result was False 2026-03-09 18:29:28.087063 | 2026-03-09 18:29:28.087227 | TASK [deploy-crc-cloud : Patch ingress config] 2026-03-09 18:29:28.103165 | crc | skipping: Conditional result was False 2026-03-09 18:29:28.118586 | 2026-03-09 18:29:28.118782 | TASK [deploy-crc-cloud : Patch api server] 2026-03-09 18:29:28.134600 | crc | skipping: Conditional result was False 2026-03-09 18:29:28.147216 | 2026-03-09 18:29:28.147396 | TASK [deploy-crc-cloud : Patch default route] 2026-03-09 18:29:28.163014 | crc | skipping: Conditional result was False 2026-03-09 18:29:28.176185 | 2026-03-09 18:29:28.176369 | TASK [deploy-crc-cloud : Wait for cluster become healthy after changing ingress api server and default route] 2026-03-09 18:29:28.205247 | crc | ok 2026-03-09 18:29:28.218050 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml 2026-03-09 18:29:28.255520 | 2026-03-09 18:29:28.255655 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy authentication|console|etcd|ingress|openshift-apiserver] 2026-03-09 18:29:28.869527 | crc | True 2026-03-09 18:29:28.870474 | crc | True 2026-03-09 18:29:28.870501 | crc | True 2026-03-09 18:29:28.870516 | crc | True 2026-03-09 18:29:28.870529 | crc | True 2026-03-09 18:29:29.300216 | crc | changed 2026-03-09 18:29:29.312537 | 2026-03-09 18:29:29.312673 | TASK [deploy-crc-cloud : Output success message if components are healthy] 2026-03-09 18:29:29.356999 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds. 2026-03-09 18:29:29.374985 | 2026-03-09 18:29:29.375217 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy] 2026-03-09 18:29:29.403838 | crc | skipping: Conditional result was False 2026-03-09 18:29:29.421299 | 2026-03-09 18:29:29.421485 | TASK [deploy-crc-cloud : Get console route] 2026-03-09 18:29:29.439237 | crc | skipping: Conditional result was False 2026-03-09 18:29:29.475264 | 2026-03-09 18:29:29.475466 | TASK [Start crc-cloud role when Zuul is not defined] 2026-03-09 18:29:29.512619 | crc | skipping: Conditional result was False 2026-03-09 18:29:29.533556 | 2026-03-09 18:29:29.533705 | TASK [prepare-crc-cloud : Recreate kubelet config.json] 2026-03-09 18:29:29.576744 | crc | ok 2026-03-09 18:29:29.598173 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/inject-pull-secret.yaml 2026-03-09 18:29:29.630652 | 2026-03-09 18:29:29.630782 | TASK [prepare-crc-cloud : Fail when openshift_pull_sec not provided] 2026-03-09 18:29:29.687357 | crc | skipping: Conditional result was False 2026-03-09 18:29:29.704757 | 2026-03-09 18:29:29.704945 | TASK [prepare-crc-cloud : Cleanup file] 2026-03-09 18:29:30.265338 | crc | Output suppressed because no_log was given 2026-03-09 18:29:30.278262 | 2026-03-09 18:29:30.278420 | TASK [prepare-crc-cloud : Copy pull-secret to kubelet config.json] 2026-03-09 18:29:30.631644 | crc | Output suppressed because no_log was given 2026-03-09 18:29:30.642702 | 2026-03-09 18:29:30.642813 | TASK [prepare-crc-cloud : Replace singleqoute with doublequote] 2026-03-09 18:29:31.138195 | crc | changed: 42 replacements made 2026-03-09 18:29:31.154904 | 2026-03-09 18:29:31.155092 | TASK [prepare-crc-cloud : Remove spaces] 2026-03-09 18:29:31.551484 | crc | changed: 20 replacements made 2026-03-09 18:29:31.562777 | 2026-03-09 18:29:31.562879 | TASK [prepare-crc-cloud : Ensure you are logged in to the OpenShift] 2026-03-09 18:29:31.587651 | crc | ok 2026-03-09 18:29:31.596183 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/openshift_login.yaml 2026-03-09 18:29:31.625507 | 2026-03-09 18:29:31.625629 | TASK [prepare-crc-cloud : Login to the OpenShift] 2026-03-09 18:29:32.035978 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported! 2026-03-09 18:29:32.149531 | crc | 2026-03-09 18:29:32.149918 | crc | Login successful. 2026-03-09 18:29:32.163620 | crc | 2026-03-09 18:29:32.163702 | crc | You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects' 2026-03-09 18:29:32.180737 | crc | 2026-03-09 18:29:32.180819 | crc | Using project "default". 2026-03-09 18:29:32.668354 | crc | ok 2026-03-09 18:29:32.686071 | 2026-03-09 18:29:32.686244 | TASK [prepare-crc-cloud : Post tasks after crc-cloud script execution] 2026-03-09 18:29:32.731672 | crc | ok 2026-03-09 18:29:32.756007 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/post.yaml 2026-03-09 18:29:32.791124 | 2026-03-09 18:29:32.791358 | LOOP [prepare-crc-cloud : Set DNS nameservers provided by cloud provider] 2026-03-09 18:29:33.146877 | crc | changed: line added 2026-03-09 18:29:33.484855 | crc | changed: line added 2026-03-09 18:29:33.501087 | 2026-03-09 18:29:33.501235 | TASK [prepare-crc-cloud : Increase dnsmasq cache size] 2026-03-09 18:29:33.902410 | crc | changed: line added 2026-03-09 18:29:33.921444 | 2026-03-09 18:29:33.921649 | TASK [prepare-crc-cloud : Reload NetworkManager after adding cache and DNS server] 2026-03-09 18:29:34.746221 | crc | changed 2026-03-09 18:29:34.757907 | 2026-03-09 18:29:34.758014 | TASK [prepare-crc-cloud : Ensure kube config does not exists] 2026-03-09 18:29:35.065997 | crc | ok 2026-03-09 18:29:35.085796 | 2026-03-09 18:29:35.085975 | TASK [prepare-crc-cloud : Copy new generated KUBECONFIG] 2026-03-09 18:29:35.105733 | crc | skipping: Conditional result was False 2026-03-09 18:29:35.127238 | 2026-03-09 18:29:35.127423 | TASK [prepare-crc-cloud : Delete all openshift-marketplace pods] 2026-03-09 18:29:35.614633 | crc | pod "certified-operators-sn8zk" deleted 2026-03-09 18:29:35.622877 | crc | pod "community-operators-nk4bg" deleted 2026-03-09 18:29:35.643600 | crc | pod "marketplace-operator-79b997595-7nw2x" deleted 2026-03-09 18:29:35.646466 | crc | pod "redhat-marketplace-nsq8f" deleted 2026-03-09 18:29:35.651926 | crc | pod "redhat-operators-2h8qw" deleted 2026-03-09 18:29:37.712941 | crc | ok 2026-03-09 18:29:37.729700 | 2026-03-09 18:29:37.729880 | TASK [prepare-crc-cloud : Create workaround for expired cert] 2026-03-09 18:29:37.795773 | crc | ok 2026-03-09 18:29:37.809677 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/cluster_cert.yaml 2026-03-09 18:29:37.848109 | 2026-03-09 18:29:37.848218 | LOOP [prepare-crc-cloud : Re-run accept certificates in Pending state few times] 2026-03-09 18:29:38.370785 | crc | ok 2026-03-09 18:29:38.371290 | crc | changed: All items complete 2026-03-09 18:29:38.371367 | 2026-03-09 18:29:48.776166 | crc | ok 2026-03-09 18:29:59.340318 | crc | ok 2026-03-09 18:29:59.364425 | 2026-03-09 18:29:59.364601 | TASK [prepare-crc-cloud : Create service account for - auto-csr-approver] 2026-03-09 18:30:00.224311 | crc | ok 2026-03-09 18:30:00.233347 | 2026-03-09 18:30:00.233456 | TASK [prepare-crc-cloud : Apply cluster cert approver] 2026-03-09 18:30:00.831954 | crc | serviceaccount/csr-approver-sa unchanged 2026-03-09 18:30:00.870275 | crc | clusterrole.rbac.authorization.k8s.io/csr-approver unchanged 2026-03-09 18:30:00.888836 | crc | clusterrolebinding.rbac.authorization.k8s.io/csr-approver-binding unchanged 2026-03-09 18:30:00.934220 | crc | cronjob.batch/auto-csr-approver unchanged 2026-03-09 18:30:01.453796 | crc | ok 2026-03-09 18:30:01.476177 | 2026-03-09 18:30:01.476407 | TASK [prepare-crc-cloud : Pause machineconfigpool] 2026-03-09 18:30:01.520850 | crc | ok 2026-03-09 18:30:01.542005 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/pause_mcp.yaml 2026-03-09 18:30:01.590295 | 2026-03-09 18:30:01.590448 | LOOP [prepare-crc-cloud : Pause the machineconfig configuration if mcp is not needed] 2026-03-09 18:30:02.102768 | crc | ok 2026-03-09 18:30:02.627745 | crc | ok 2026-03-09 18:30:02.645488 | 2026-03-09 18:30:02.645675 | TASK [prepare-crc-cloud : Print current machineconfigpool] 2026-03-09 18:30:03.182372 | crc | NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE 2026-03-09 18:30:03.194409 | crc | master rendered-master-f83a149c7490c3b0bf850df839837383 False True True 1 0 0 1 379d 2026-03-09 18:30:03.194542 | crc | worker rendered-worker-6162d862e9ce08916fb3139996b7f663 True False False 0 0 0 0 379d 2026-03-09 18:30:03.373683 | crc | ok 2026-03-09 18:30:03.384880 | 2026-03-09 18:30:03.385048 | TASK [prepare-crc-cloud : Set low profile to etcd to avoid flakiness] 2026-03-09 18:30:03.411084 | crc | skipping: Conditional result was False 2026-03-09 18:30:03.426157 | 2026-03-09 18:30:03.426351 | TASK [prepare-crc-cloud : Fix machineconfigpool degradation] 2026-03-09 18:30:03.463180 | crc | skipping: Conditional result was False 2026-03-09 18:30:03.475284 | 2026-03-09 18:30:03.475442 | TASK [prepare-crc-cloud : Remove drain machineconfig for the node] 2026-03-09 18:30:03.501582 | crc | skipping: Conditional result was False 2026-03-09 18:30:03.516785 | 2026-03-09 18:30:03.516958 | TASK [prepare-crc-cloud : Post machineconfigpool tasks] 2026-03-09 18:30:03.543870 | crc | skipping: Conditional result was False 2026-03-09 18:30:03.557283 | 2026-03-09 18:30:03.557445 | TASK [prepare-crc-cloud : Enable monitoring and cluster-monitoring-operator] 2026-03-09 18:30:03.585450 | crc | skipping: Conditional result was False 2026-03-09 18:30:03.611148 | 2026-03-09 18:30:03.611322 | TASK [Run crc-extracted preparation] 2026-03-09 18:30:03.638044 | crc | skipping: Conditional result was False 2026-03-09 18:30:03.652700 | 2026-03-09 18:30:03.652873 | TASK [Fail when image build date does not exists] 2026-03-09 18:30:03.679789 | crc | skipping: Conditional result was False 2026-03-09 18:30:03.708010 | 2026-03-09 18:30:03.708139 | PLAY [Set IOPS and Read/Write disk limitation] 2026-03-09 18:30:03.737452 | 2026-03-09 18:30:03.737617 | TASK [Set limit for disk usage - IOPS and Read/Write] 2026-03-09 18:30:03.811919 | controller | ok 2026-03-09 18:30:03.828397 | crc | ok 2026-03-09 18:30:03.899513 | 2026-03-09 18:30:03.899634 | TASK [disk-usage-limitation : Get block device major:minor] 2026-03-09 18:30:04.305121 | crc | 252:0 2026-03-09 18:30:04.410964 | controller | 252:0 2026-03-09 18:30:04.467122 | crc | ok 2026-03-09 18:30:04.475815 | controller | ok 2026-03-09 18:30:04.484794 | 2026-03-09 18:30:04.485111 | LOOP [disk-usage-limitation : Ensure that some cgroup dirs exists] 2026-03-09 18:30:04.739644 | controller | ok: "/sys/fs/cgroup/init.scope" 2026-03-09 18:30:04.740197 | controller | ok: All items complete 2026-03-09 18:30:04.740288 | 2026-03-09 18:30:04.812768 | crc | ok: "/sys/fs/cgroup/init.scope" 2026-03-09 18:30:04.956936 | controller | changed: "/sys/fs/cgroup/machine.slice" 2026-03-09 18:30:05.152293 | controller | ok: "/sys/fs/cgroup/system.slice" 2026-03-09 18:30:05.375058 | controller | ok: "/sys/fs/cgroup/user.slice" 2026-03-09 18:30:05.541225 | crc | ok: "/sys/fs/cgroup/machine.slice" 2026-03-09 18:30:06.434107 | crc | ok: "/sys/fs/cgroup/system.slice" 2026-03-09 18:30:06.799551 | crc | ok: "/sys/fs/cgroup/user.slice" 2026-03-09 18:30:06.817580 | 2026-03-09 18:30:06.817795 | TASK [disk-usage-limitation : Ensure systemd configuration directory] 2026-03-09 18:30:07.050233 | controller | changed 2026-03-09 18:30:07.151803 | crc | ok 2026-03-09 18:30:07.167624 | 2026-03-09 18:30:07.167845 | TASK [disk-usage-limitation : Enable IO limit via override configuration] 2026-03-09 18:30:07.828302 | controller | changed 2026-03-09 18:30:08.046842 | crc | changed 2026-03-09 18:30:08.060610 | 2026-03-09 18:30:08.060770 | TASK [disk-usage-limitation : Flush handlers that will call systemd daemon-reload] 2026-03-09 18:30:08.078301 | 2026-03-09 18:30:08.078426 | TASK [disk-usage-limitation : Flush handlers that will call systemd daemon-reload] 2026-03-09 18:30:08.096437 | 2026-03-09 18:30:08.096570 | TASK [disk-usage-limitation : Reload systemd daemon after enabling IOAccounting] 2026-03-09 18:30:09.029793 | controller | ok 2026-03-09 18:30:10.212893 | crc | ok 2026-03-09 18:30:10.226335 | 2026-03-09 18:30:10.226452 | TASK [disk-usage-limitation : Wait for system.slice cgroup to have io.max file] 2026-03-09 18:30:10.496094 | controller | ok 2026-03-09 18:30:10.672348 | crc | ok 2026-03-09 18:30:10.681733 | 2026-03-09 18:30:10.681867 | LOOP [disk-usage-limitation : Set IOPS limit in the root cgroup] 2026-03-09 18:30:10.960859 | controller | ok 2026-03-09 18:30:10.961409 | controller | ok: All items complete 2026-03-09 18:30:10.961474 | 2026-03-09 18:30:11.066742 | crc | ok 2026-03-09 18:30:11.211639 | controller | ok 2026-03-09 18:30:11.445011 | controller | ok 2026-03-09 18:30:11.470294 | crc | ok 2026-03-09 18:30:11.642473 | controller | ok 2026-03-09 18:30:11.769889 | crc | ok 2026-03-09 18:30:12.095142 | crc | ok 2026-03-09 18:30:12.114146 | 2026-03-09 18:30:12.114455 | TASK [disk-usage-limitation : Verify the IOPS limit] 2026-03-09 18:30:12.497012 | crc | init 2026-03-09 18:30:12.498739 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:12.499924 | crc | machine 2026-03-09 18:30:12.499946 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:12.500134 | crc | system 2026-03-09 18:30:12.502300 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:12.502699 | crc | user 2026-03-09 18:30:12.504142 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:12.430274 | controller | init 2026-03-09 18:30:12.430332 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:12.430343 | controller | machine 2026-03-09 18:30:12.430350 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:12.430357 | controller | system 2026-03-09 18:30:12.430363 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:12.430370 | controller | user 2026-03-09 18:30:12.430376 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:12.708950 | controller | ok 2026-03-09 18:30:12.725984 | crc | ok 2026-03-09 18:30:12.740892 | 2026-03-09 18:30:12.741110 | TASK [disk-usage-limitation : Check if kubepods.slice exists] 2026-03-09 18:30:13.047890 | crc | ok 2026-03-09 18:30:13.073301 | controller | ok 2026-03-09 18:30:13.092373 | 2026-03-09 18:30:13.092975 | TASK [disk-usage-limitation : Set the limit] 2026-03-09 18:30:13.677126 | controller | skipping: Conditional result was False 2026-03-09 18:30:13.694655 | crc | ok 2026-03-09 18:30:13.709920 | 2026-03-09 18:30:13.710152 | TASK [disk-usage-limitation : Verify the IOPS limit - kubepods slice] 2026-03-09 18:30:14.129259 | crc | kubepods 2026-03-09 18:30:14.130521 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.347040 | controller | skipping: Conditional result was False 2026-03-09 18:30:14.355422 | crc | ok 2026-03-09 18:30:14.361973 | 2026-03-09 18:30:14.362061 | TASK [disk-usage-limitation : Print the current io.max value] 2026-03-09 18:30:14.432071 | controller | ok: 2026-03-09 18:30:14.432362 | controller | init 2026-03-09 18:30:14.432403 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.432431 | controller | machine 2026-03-09 18:30:14.432457 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.432481 | controller | system 2026-03-09 18:30:14.432505 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.432528 | controller | user 2026-03-09 18:30:14.432551 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.464890 | crc | ok: 2026-03-09 18:30:14.465049 | crc | init 2026-03-09 18:30:14.465090 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.465118 | crc | machine 2026-03-09 18:30:14.465142 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.465166 | crc | system 2026-03-09 18:30:14.465189 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.465212 | crc | user 2026-03-09 18:30:14.465234 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 kubepods 2026-03-09 18:30:14.465257 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-09 18:30:14.547908 | 2026-03-09 18:30:14.548075 | PLAY RECAP 2026-03-09 18:30:14.548126 | controller | ok: 10 changed: 3 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0 2026-03-09 18:30:14.548152 | crc | ok: 112 changed: 37 unreachable: 0 failed: 0 skipped: 28 rescued: 0 ignored: 0 2026-03-09 18:30:14.548169 | 2026-03-09 18:30:14.689585 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/prepare-crc.yaml@master] 2026-03-09 18:30:14.694275 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/podman.yaml@master] 2026-03-09 18:30:15.259160 | 2026-03-09 18:30:15.259363 | PLAY [all,!compute] 2026-03-09 18:30:15.295331 | 2026-03-09 18:30:15.295466 | TASK [Enable container-tools rhel-modules for c8/crc node] 2026-03-09 18:30:15.857283 | controller | skipping: Conditional result was False 2026-03-09 18:30:15.858378 | crc | skipping: Conditional result was False 2026-03-09 18:30:15.872718 | 2026-03-09 18:30:15.872929 | TASK [Install podman] 2026-03-09 18:30:15.956134 | crc | skipping: Conditional result was False 2026-03-09 18:31:09.645191 | controller | changed 2026-03-09 18:31:09.661589 | 2026-03-09 18:31:09.661797 | TASK [Set var name for quay login zuul secret] 2026-03-09 18:31:09.742163 | crc | skipping: Conditional result was False 2026-03-09 18:31:09.742818 | controller | skipping: Conditional result was False 2026-03-09 18:31:09.752372 | 2026-03-09 18:31:09.752497 | TASK [Print the username] 2026-03-09 18:31:10.202656 | controller | openstack-k8s-operators+cirobot 2026-03-09 18:31:10.343473 | crc | skipping: Conditional result was False 2026-03-09 18:31:10.353690 | controller | ok: Runtime: 0:00:00.003871 2026-03-09 18:31:10.363573 | 2026-03-09 18:31:10.363709 | TASK [Fail when user or password is not set] 2026-03-09 18:31:10.440468 | controller | skipping: Conditional result was False 2026-03-09 18:31:10.461910 | crc | skipping: Conditional result was False 2026-03-09 18:31:10.471857 | 2026-03-09 18:31:10.472010 | TASK [Set vars for quay login] 2026-03-09 18:31:10.558508 | controller | Output suppressed because no_log was given 2026-03-09 18:31:10.567419 | 2026-03-09 18:31:10.567517 | LOOP [Perform container registry login(s) with podman] 2026-03-09 18:31:11.454664 | controller | Output suppressed because no_log was given 2026-03-09 18:31:11.467467 | 2026-03-09 18:31:11.467648 | LOOP [Perform container registry login(s) with buildah] 2026-03-09 18:31:11.554363 | controller | Output suppressed because no_log was given 2026-03-09 18:31:11.554846 | 2026-03-09 18:31:11.603749 | 2026-03-09 18:31:11.603955 | TASK [Set Insecure registry for content provider] 2026-03-09 18:31:11.695937 | crc | skipping: Conditional result was False 2026-03-09 18:31:12.100742 | controller | changed: Block inserted 2026-03-09 18:31:12.160101 | 2026-03-09 18:31:12.160280 | PLAY RECAP 2026-03-09 18:31:12.160349 | controller | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0 2026-03-09 18:31:12.160383 | crc | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 9 rescued: 0 ignored: 0 2026-03-09 18:31:12.160401 | 2026-03-09 18:31:12.277671 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/podman.yaml@master] 2026-03-09 18:31:12.287999 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/data_plane_adoption/dpa_RH_subscription_registry_login.yaml@master] 2026-03-09 18:31:12.939460 | 2026-03-09 18:31:12.939580 | PLAY [rh-subscription] 2026-03-09 18:31:12.939668 | 2026-03-09 18:31:12.939702 | PLAY RECAP 2026-03-09 18:31:12.939730 | 2026-03-09 18:31:13.019818 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/data_plane_adoption/dpa_RH_subscription_registry_login.yaml@master] 2026-03-09 18:31:13.024064 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-customizations.yml@main] 2026-03-09 18:31:13.711463 | 2026-03-09 18:31:13.711591 | PLAY [Tweak CRC node] 2026-03-09 18:31:13.740708 | 2026-03-09 18:31:13.740864 | TASK [Load network parameters] 2026-03-09 18:31:14.424301 | crc | ok 2026-03-09 18:31:14.434120 | 2026-03-09 18:31:14.434301 | TASK [Check which dnsmasq config we must edit] 2026-03-09 18:31:14.887114 | crc | ok 2026-03-09 18:31:14.943389 | 2026-03-09 18:31:14.943554 | TASK [Ensure crc knows about its second NIC] 2026-03-09 18:31:15.830815 | crc | changed 2026-03-09 18:31:15.849456 | 2026-03-09 18:31:15.849696 | TASK [Ensure crc does not get "public" DNS] 2026-03-09 18:31:16.542472 | crc | changed 2026-03-09 18:31:16.559973 | 2026-03-09 18:31:16.560200 | TASK [Restart NetworkManager] 2026-03-09 18:31:17.678444 | crc | changed 2026-03-09 18:31:17.687572 | 2026-03-09 18:31:17.687645 | LOOP [Configure dns forwarders] 2026-03-09 18:31:18.170978 | crc | ok 2026-03-09 18:31:18.420276 | crc | ok 2026-03-09 18:31:18.446625 | 2026-03-09 18:31:18.446793 | TASK [Configure local DNS for CRC pod] 2026-03-09 18:31:18.902791 | crc | ok 2026-03-09 18:31:18.913650 | 2026-03-09 18:31:18.913782 | TASK [Configure dnsmasq listen-address to listen on both br-ex and ci-private-network] 2026-03-09 18:31:19.292069 | crc | changed: line added 2026-03-09 18:31:19.309208 | 2026-03-09 18:31:19.309372 | TASK [Restart native dnsmasq service] 2026-03-09 18:31:19.877963 | crc | changed 2026-03-09 18:31:19.896903 | 2026-03-09 18:31:19.897080 | TASK [Stop dnsmasq] 2026-03-09 18:31:19.922710 | crc | skipping: Conditional result was False 2026-03-09 18:31:19.931478 | 2026-03-09 18:31:19.931594 | TASK [Make sure that crc-dnsmasq is not running] 2026-03-09 18:31:19.956149 | crc | skipping: Conditional result was False 2026-03-09 18:31:19.964865 | 2026-03-09 18:31:19.964983 | TASK [Start dnsmasq] 2026-03-09 18:31:19.989460 | crc | skipping: Conditional result was False 2026-03-09 18:31:19.999924 | 2026-03-09 18:31:20.000084 | TASK [Wait for CRC to be ready] 2026-03-09 18:31:20.680084 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported! 2026-03-09 18:31:20.856999 | crc | 2026-03-09 18:31:20.857082 | crc | Login successful. 2026-03-09 18:31:20.896115 | crc | 2026-03-09 18:31:20.896176 | crc | You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects' 2026-03-09 18:31:20.917768 | crc | 2026-03-09 18:31:20.917915 | crc | Using project "default". 2026-03-09 18:31:21.042995 | crc | changed 2026-03-09 18:31:21.059816 | 2026-03-09 18:31:21.059962 | TASK [Set insecure registry on crc node] 2026-03-09 18:31:21.115228 | crc | ok 2026-03-09 18:31:21.138581 | crc | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/ci/playbooks/tasks/set_crc_insecure_registry.yml 2026-03-09 18:31:21.149801 | 2026-03-09 18:31:21.149895 | TASK [Patch the image.config.openshift.io resource to include insecure registry] 2026-03-09 18:31:21.700543 | crc | image.config.openshift.io/cluster patched 2026-03-09 18:31:22.197685 | crc | changed 2026-03-09 18:31:22.205384 | 2026-03-09 18:31:22.205497 | TASK [Patch the image.config.openshift.io resource to allow registries] 2026-03-09 18:31:22.756788 | crc | image.config.openshift.io/cluster patched 2026-03-09 18:31:22.816767 | crc | changed 2026-03-09 18:31:22.834950 | 2026-03-09 18:31:22.835116 | LOOP [Add additional allowed registries] 2026-03-09 18:31:22.894929 | 2026-03-09 18:31:22.895192 | TASK [Ensure registries.conf.d exists] 2026-03-09 18:31:23.310331 | crc | ok 2026-03-09 18:31:23.319092 | 2026-03-09 18:31:23.319157 | TASK [Set Insecure registry for content provider] 2026-03-09 18:31:23.748374 | crc | changed: File created 2026-03-09 18:31:23.765389 | 2026-03-09 18:31:23.765616 | TASK [Set registry mirror override] 2026-03-09 18:31:23.804322 | crc | skipping: Conditional result was False 2026-03-09 18:31:23.822666 | 2026-03-09 18:31:23.822865 | TASK [Restart crio] 2026-03-09 18:31:25.439864 | crc | changed 2026-03-09 18:31:25.471579 | 2026-03-09 18:31:25.471679 | PLAY [Tweak Controller] 2026-03-09 18:31:25.491396 | 2026-03-09 18:31:25.491526 | TASK [Install other packages] 2026-03-09 18:31:29.206438 | controller | changed 2026-03-09 18:31:29.212942 | 2026-03-09 18:31:29.213050 | LOOP [Ensure we know compute host keys] 2026-03-09 18:31:29.624450 | controller | changed 2026-03-09 18:31:29.624859 | 2026-03-09 18:31:29.886518 | controller | changed 2026-03-09 18:31:29.905344 | 2026-03-09 18:31:29.905516 | TASK [Generate an ssh keypair] 2026-03-09 18:31:30.198192 | controller | Generating public/private ecdsa key pair. 2026-03-09 18:31:30.198605 | controller | Your identification has been saved in /home/zuul/.ssh/id_cifw 2026-03-09 18:31:30.198617 | controller | Your public key has been saved in /home/zuul/.ssh/id_cifw.pub 2026-03-09 18:31:30.198622 | controller | The key fingerprint is: 2026-03-09 18:31:30.198627 | controller | SHA256:5ivMolkGYXwb3RzOobm2ApCnVhwGvwipYGOmuUrfxEM zuul@np0005642801.novalocal 2026-03-09 18:31:30.198632 | controller | The key's randomart image is: 2026-03-09 18:31:30.198636 | controller | +---[ECDSA 256]---+ 2026-03-09 18:31:30.198641 | controller | | ..o o | 2026-03-09 18:31:30.198645 | controller | | ++ .. B o | 2026-03-09 18:31:30.198650 | controller | |*==+o + = | 2026-03-09 18:31:30.198654 | controller | |O*+o.o . | 2026-03-09 18:31:30.198658 | controller | |=oo..Eo S | 2026-03-09 18:31:30.198663 | controller | |.. oo. + | 2026-03-09 18:31:30.198667 | controller | |.. +*. . | 2026-03-09 18:31:30.198671 | controller | |o .++.= . | 2026-03-09 18:31:30.198675 | controller | |. oo o .. | 2026-03-09 18:31:30.198680 | controller | +----[SHA256]-----+ 2026-03-09 18:31:30.460874 | controller | changed 2026-03-09 18:31:30.470108 | 2026-03-09 18:31:30.470207 | TASK [Get public key] 2026-03-09 18:31:30.779264 | controller | ok 2026-03-09 18:31:30.793001 | 2026-03-09 18:31:30.793264 | TASK [Register pub key as a fact] 2026-03-09 18:31:30.869614 | controller | ok 2026-03-09 18:31:30.878340 | 2026-03-09 18:31:30.878458 | TASK [Load network parameters] 2026-03-09 18:31:31.154852 | controller | ok 2026-03-09 18:31:31.174057 | 2026-03-09 18:31:31.174271 | TASK [Check if cloud-init is overriding NM settings] 2026-03-09 18:31:31.495061 | controller | ok 2026-03-09 18:31:31.501326 | 2026-03-09 18:31:31.501390 | TASK [Remove cloud-init DNS override if present] 2026-03-09 18:31:31.527392 | controller | skipping: Conditional result was False 2026-03-09 18:31:31.579662 | 2026-03-09 18:31:31.579786 | TASK [Reload the NetworkManager to pick the changes] 2026-03-09 18:31:31.608939 | controller | skipping: Conditional result was False 2026-03-09 18:31:31.617262 | 2026-03-09 18:31:31.617387 | TASK [Get the default iface connection] 2026-03-09 18:31:31.922238 | controller | System eth0 2026-03-09 18:31:32.152714 | controller | changed 2026-03-09 18:31:32.169401 | 2026-03-09 18:31:32.169605 | TASK [Prepend CRC DNS server in the controllers default Network Manager connection configuation] 2026-03-09 18:31:32.630886 | controller | Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5) 2026-03-09 18:31:32.706047 | controller | changed 2026-03-09 18:31:32.740766 | 2026-03-09 18:31:32.740878 | PLAY [Ensure persistent SSH access] 2026-03-09 18:31:32.773727 | 2026-03-09 18:31:32.773864 | TASK [Inject key in ansible_user] 2026-03-09 18:31:33.240364 | controller | changed 2026-03-09 18:31:33.767659 | crc | changed 2026-03-09 18:31:33.780107 | 2026-03-09 18:31:33.780240 | TASK [Inject key in root] 2026-03-09 18:31:34.136666 | controller | changed 2026-03-09 18:31:34.259811 | crc | changed 2026-03-09 18:31:34.324907 | 2026-03-09 18:31:34.325100 | PLAY [Add cloud-admin user on Compute] 2026-03-09 18:31:34.326271 | 2026-03-09 18:31:34.326348 | PLAY [Set some fancy hostname] 2026-03-09 18:31:34.342977 | 2026-03-09 18:31:34.343060 | TASK [Inject hostname in configuration file] 2026-03-09 18:31:35.114747 | controller | changed 2026-03-09 18:31:35.166296 | 2026-03-09 18:31:35.166454 | PLAY RECAP 2026-03-09 18:31:35.166545 | controller | ok: 12 changed: 8 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0 2026-03-09 18:31:35.166604 | crc | ok: 18 changed: 12 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-09 18:31:35.166642 | 2026-03-09 18:31:35.301729 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-customizations.yml@main] 2026-03-09 18:31:35.306179 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-prepare.yml@main] 2026-03-09 18:31:35.969225 | 2026-03-09 18:31:35.969345 | PLAY [Run ci/playbooks/e2e-prepare.yml] 2026-03-09 18:31:35.993528 | 2026-03-09 18:31:35.993653 | TASK [Clone repos in the job workspace] 2026-03-09 18:31:36.013978 | controller | ok 2026-03-09 18:31:36.032380 | 2026-03-09 18:31:36.032489 | TASK [prepare-workspace : Start zuul_console daemon.] 2026-03-09 18:31:36.547971 | controller | ok 2026-03-09 18:31:36.562698 | 2026-03-09 18:31:36.562864 | TASK [prepare-workspace : Synchronize src repos to workspace directory.] 2026-03-09 18:31:50.855027 | controller | Output suppressed because no_log was given 2026-03-09 18:31:50.866556 | 2026-03-09 18:31:50.866674 | TASK [Create zuul-output directory] 2026-03-09 18:31:51.251699 | controller | ok 2026-03-09 18:31:51.267799 | 2026-03-09 18:31:51.267972 | TASK [Install required packages] 2026-03-09 18:31:52.366557 | controller | ok: Nothing to do 2026-03-09 18:31:52.382062 | 2026-03-09 18:31:52.382230 | TASK [Output pip related things] 2026-03-09 18:31:53.071805 | controller | pip 21.3.1 from /usr/lib/python3.9/site-packages/pip (python 3.9) 2026-03-09 18:31:53.463830 | controller | changed 2026-03-09 18:31:53.472667 | 2026-03-09 18:31:53.472791 | TASK [Install requirements] 2026-03-09 18:32:57.296318 | controller | changed 2026-03-09 18:32:57.317930 | 2026-03-09 18:32:57.318069 | TASK [Add cronjob to trigger job stats] 2026-03-09 18:32:57.363257 | controller | skipping: Conditional result was False 2026-03-09 18:32:57.372863 | 2026-03-09 18:32:57.372976 | LOOP [Construct project change list] 2026-03-09 18:32:57.428301 | controller | ok: 2026-03-09 18:32:57.428512 | controller | { 2026-03-09 18:32:57.428540 | controller | "branch": "master", 2026-03-09 18:32:57.428559 | controller | "change": "979276", 2026-03-09 18:32:57.428576 | controller | "change_message": "Ensure zone migration creates the correct action\n\nFix the logic in the zone migration strategy that decides whether to\nmigrate or retype a volume. After a change in the schema[1], the zone\nmigration strategy does not support the user passing both dst_pool\nand dst_type in the same input pool.\n\nWith the current change, if dst_type is not passed, only\nmigration action should be created, never retypes.\n\n[1] https://review.opendev.org/c/openstack/watcher/+/964776\n\nChange-Id: I4c3f90eab927e4c5c902f610cb751ba0eb65001e\nSigned-off-by: jgilaber \n", 2026-03-09 18:32:57.428605 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher/+/979276", 2026-03-09 18:32:57.428628 | controller | "commit_id": "64164197e0a8ae9e5016567724b13e7a66130ae7", 2026-03-09 18:32:57.428646 | controller | "patchset": "3", 2026-03-09 18:32:57.428663 | controller | "project": { 2026-03-09 18:32:57.428680 | controller | "canonical_hostname": "opendev.org", 2026-03-09 18:32:57.428696 | controller | "canonical_name": "opendev.org/openstack/watcher", 2026-03-09 18:32:57.428713 | controller | "name": "openstack/watcher", 2026-03-09 18:32:57.428823 | controller | "short_name": "watcher", 2026-03-09 18:32:57.428849 | controller | "src_dir": "src/opendev.org/openstack/watcher" 2026-03-09 18:32:57.428869 | controller | }, 2026-03-09 18:32:57.428886 | controller | "topic": null 2026-03-09 18:32:57.428903 | controller | } 2026-03-09 18:32:57.436812 | controller | ok: 2026-03-09 18:32:57.436920 | controller | { 2026-03-09 18:32:57.436945 | controller | "branch": "master", 2026-03-09 18:32:57.436965 | controller | "change": "979643", 2026-03-09 18:32:57.436983 | controller | "change_message": "Enable SameBackendFilter in cinder scheduler\n\nAdd SameBackendFilter to cinder scheduler_default_filters\nin the zuul job configuration to control volume scheduling\nduring migration tests.\n\nChange-Id: Ie9c4391a0ab80055f31c1befeaf5383d03dc4621\nSigned-off-by: jgilaber \n", 2026-03-09 18:32:57.437004 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher/+/979643", 2026-03-09 18:32:57.437098 | controller | "commit_id": "29e4137808ab2295c733b36b9851bdace3dec60a", 2026-03-09 18:32:57.437120 | controller | "patchset": "3", 2026-03-09 18:32:57.437137 | controller | "project": { 2026-03-09 18:32:57.437152 | controller | "canonical_hostname": "opendev.org", 2026-03-09 18:32:57.437168 | controller | "canonical_name": "opendev.org/openstack/watcher", 2026-03-09 18:32:57.437194 | controller | "name": "openstack/watcher", 2026-03-09 18:32:57.437212 | controller | "short_name": "watcher", 2026-03-09 18:32:57.437229 | controller | "src_dir": "src/opendev.org/openstack/watcher" 2026-03-09 18:32:57.437246 | controller | }, 2026-03-09 18:32:57.437263 | controller | "topic": null 2026-03-09 18:32:57.437281 | controller | } 2026-03-09 18:32:57.447928 | controller | ok: 2026-03-09 18:32:57.448074 | controller | { 2026-03-09 18:32:57.448104 | controller | "branch": "master", 2026-03-09 18:32:57.448124 | controller | "change": "958644", 2026-03-09 18:32:57.448141 | controller | "change_message": "Add test for volume migrate with zone migration\n\nAdd a test that creates a zone migration audit and migrates both an\navailable and attached volume. The test checks that there is more than\none cinder pool to be able to migrate volumes between them, but makes no\nassertion on the backend configuration.\n\nDepends-On: https://review.opendev.org/c/openstack/watcher/+/979276\nDepends-On: https://review.opendev.org/c/openstack/watcher/+/979643\n\nAssited-By: Cursor (Claude-4-sonnet)\nChange-Id: I4e7e103109f830e28961856dfc3c197d89347e63\nSigned-off-by: jgilaber \n", 2026-03-09 18:32:57.448306 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/958644", 2026-03-09 18:32:57.448336 | controller | "commit_id": "5931d8d6e69fb6fa67320bbc508b3744b12bed85", 2026-03-09 18:32:57.448355 | controller | "patchset": "32", 2026-03-09 18:32:57.448372 | controller | "project": { 2026-03-09 18:32:57.448388 | controller | "canonical_hostname": "opendev.org", 2026-03-09 18:32:57.448405 | controller | "canonical_name": "opendev.org/openstack/watcher-tempest-plugin", 2026-03-09 18:32:57.448422 | controller | "name": "openstack/watcher-tempest-plugin", 2026-03-09 18:32:57.448438 | controller | "short_name": "watcher-tempest-plugin", 2026-03-09 18:32:57.448461 | controller | "src_dir": "src/opendev.org/openstack/watcher-tempest-plugin" 2026-03-09 18:32:57.448486 | controller | }, 2026-03-09 18:32:57.448509 | controller | "topic": null 2026-03-09 18:32:57.448531 | controller | } 2026-03-09 18:32:57.457382 | controller | ok: 2026-03-09 18:32:57.457488 | controller | { 2026-03-09 18:32:57.457511 | controller | "branch": "master", 2026-03-09 18:32:57.457529 | controller | "change": "962702", 2026-03-09 18:32:57.457549 | controller | "change_message": "Test zone migration volume and compute migrations\n\nReplace the volume migration test with a new one that verifies zone\nmigration strategy can migrate both volumes and compute instances\nin the same action plan.\n\nGenerated-By: Cursor (claude-4-sonnet)\nChange-Id: I748d1316356e6ad116791eb33c3451c2a9c41636\nSigned-off-by: jgilaber \n", 2026-03-09 18:32:57.457570 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/962702", 2026-03-09 18:32:57.457587 | controller | "commit_id": "b0c5f8ac8fc6c508dc18eab2816d38f4c959083b", 2026-03-09 18:32:57.457603 | controller | "patchset": "25", 2026-03-09 18:32:57.457619 | controller | "project": { 2026-03-09 18:32:57.457634 | controller | "canonical_hostname": "opendev.org", 2026-03-09 18:32:57.457649 | controller | "canonical_name": "opendev.org/openstack/watcher-tempest-plugin", 2026-03-09 18:32:57.457665 | controller | "name": "openstack/watcher-tempest-plugin", 2026-03-09 18:32:57.457680 | controller | "short_name": "watcher-tempest-plugin", 2026-03-09 18:32:57.457696 | controller | "src_dir": "src/opendev.org/openstack/watcher-tempest-plugin" 2026-03-09 18:32:57.457718 | controller | }, 2026-03-09 18:32:57.457736 | controller | "topic": null 2026-03-09 18:32:57.457752 | controller | } 2026-03-09 18:32:57.466648 | controller | ok: 2026-03-09 18:32:57.466711 | controller | { 2026-03-09 18:32:57.466733 | controller | "branch": "master", 2026-03-09 18:32:57.466750 | controller | "change": "963559", 2026-03-09 18:32:57.466768 | controller | "change_message": "Add extra checks to zone migration retype test\n\nAdd two extra checks to the zone migration test covering volume retype\nto prevent regressions of two bugs:\n\n1. Ensure the src_type parameter filter volumes to retype[1]\n2. Ensure that the audit does not fail if the 'with_attached_volume'\n parameter is passed to an audit without 'compute_nodes'[2]\n\n[1] https://bugs.launchpad.net/watcher/+bug/2111507\n[2] https://bugs.launchpad.net/watcher/+bug/2111429\n\nChange-Id: I3987230d384b509921ded80777b4d5b7c407ba85\nSigned-off-by: jgilaber \n", 2026-03-09 18:32:57.466786 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559", 2026-03-09 18:32:57.466802 | controller | "commit_id": "6fedf196e61ec8ee8c9655e8fa463f714bd0ce13", 2026-03-09 18:32:57.466818 | controller | "patchset": "16", 2026-03-09 18:32:57.466834 | controller | "project": { 2026-03-09 18:32:57.466850 | controller | "canonical_hostname": "opendev.org", 2026-03-09 18:32:57.466866 | controller | "canonical_name": "opendev.org/openstack/watcher-tempest-plugin", 2026-03-09 18:32:57.466881 | controller | "name": "openstack/watcher-tempest-plugin", 2026-03-09 18:32:57.466896 | controller | "short_name": "watcher-tempest-plugin", 2026-03-09 18:32:57.466912 | controller | "src_dir": "src/opendev.org/openstack/watcher-tempest-plugin" 2026-03-09 18:32:57.466927 | controller | }, 2026-03-09 18:32:57.466942 | controller | "topic": null 2026-03-09 18:32:57.466957 | controller | } 2026-03-09 18:32:57.475440 | controller | ok: 2026-03-09 18:32:57.475539 | controller | { 2026-03-09 18:32:57.475562 | controller | "branch": "main", 2026-03-09 18:32:57.475580 | controller | "change": "351", 2026-03-09 18:32:57.475597 | controller | "change_message": "Enable SameBackendFilter in cinder scheduler\n\nAdd SameBackendFilter to cinder scheduler_default_filters\nin the nfs job configuration to control volume scheduling\nduring migration tests.\n\nDepends-On: https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559\n", 2026-03-09 18:32:57.475615 | controller | "change_url": "https://github.com/openstack-k8s-operators/watcher-operator/pull/351", 2026-03-09 18:32:57.475632 | controller | "commit_id": "2e035aad6e396aeb72cc6aec8684c43e59f8b674", 2026-03-09 18:32:57.475649 | controller | "patchset": "2e035aad6e396aeb72cc6aec8684c43e59f8b674", 2026-03-09 18:32:57.475665 | controller | "project": { 2026-03-09 18:32:57.475681 | controller | "canonical_hostname": "github.com", 2026-03-09 18:32:57.475697 | controller | "canonical_name": "github.com/openstack-k8s-operators/watcher-operator", 2026-03-09 18:32:57.475713 | controller | "name": "openstack-k8s-operators/watcher-operator", 2026-03-09 18:32:57.475729 | controller | "short_name": "watcher-operator", 2026-03-09 18:32:57.475745 | controller | "src_dir": "src/github.com/openstack-k8s-operators/watcher-operator" 2026-03-09 18:32:57.475764 | controller | }, 2026-03-09 18:32:57.475782 | controller | "topic": null 2026-03-09 18:32:57.475798 | controller | } 2026-03-09 18:32:57.527849 | 2026-03-09 18:32:57.527943 | PLAY [Run ci/playbooks/build_runner_image.yml] 2026-03-09 18:32:57.557617 | 2026-03-09 18:32:57.557717 | TASK [Filter out host if needed] 2026-03-09 18:32:57.568761 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller 2026-03-09 18:32:57.574876 | 2026-03-09 18:32:57.574939 | TASK [Filter out host if needed] 2026-03-09 18:32:57.583389 | crc | skipping: end_host conditional evaluated to False, continuing execution for crc 2026-03-09 18:32:57.588735 | 2026-03-09 18:32:57.588791 | TASK [Get git tag for image tagging] 2026-03-09 18:32:58.181659 | controller | skipping: Conditional result was False 2026-03-09 18:32:58.182825 | crc | skipping: Conditional result was False 2026-03-09 18:32:58.198006 | 2026-03-09 18:32:58.198204 | TASK [Set openstack-runner image] 2026-03-09 18:32:58.236722 | controller | skipping: Conditional result was False 2026-03-09 18:32:58.247777 | crc | skipping: Conditional result was False 2026-03-09 18:32:58.256910 | 2026-03-09 18:32:58.257066 | TASK [Build openstack-runner image] 2026-03-09 18:32:58.279795 | controller | skipping: Conditional result was False 2026-03-09 18:32:58.291271 | crc | skipping: Conditional result was False 2026-03-09 18:32:58.302798 | 2026-03-09 18:32:58.302951 | TASK [Push openstack-runner image] 2026-03-09 18:32:58.323520 | controller | skipping: Conditional result was False 2026-03-09 18:32:58.334992 | crc | skipping: Conditional result was False 2026-03-09 18:32:58.341754 | 2026-03-09 18:32:58.341837 | TASK [Make sure ci-framework directory exists] 2026-03-09 18:32:58.368423 | controller | skipping: Conditional result was False 2026-03-09 18:32:58.369053 | crc | skipping: Conditional result was False 2026-03-09 18:32:58.376222 | 2026-03-09 18:32:58.376298 | TASK [Create EDPM ansible var file] 2026-03-09 18:32:58.405634 | controller | skipping: Conditional result was False 2026-03-09 18:32:58.406319 | crc | skipping: Conditional result was False 2026-03-09 18:32:58.452444 | 2026-03-09 18:32:58.452546 | PLAY RECAP 2026-03-09 18:32:58.452591 | controller | ok: 7 changed: 3 unreachable: 0 failed: 0 skipped: 7 rescued: 0 ignored: 0 2026-03-09 18:32:58.452617 | crc | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 6 rescued: 0 ignored: 0 2026-03-09 18:32:58.452635 | 2026-03-09 18:32:58.563764 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-prepare.yml@main] 2026-03-09 18:32:58.567917 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/dump_zuul_data.yml@main] 2026-03-09 18:32:59.168447 | 2026-03-09 18:32:59.168564 | PLAY [Run ci/playbooks/dump_zuul_data.yml] 2026-03-09 18:32:59.192298 | 2026-03-09 18:32:59.192413 | TASK [Create zuul-output directory] 2026-03-09 18:32:59.762653 | controller | changed 2026-03-09 18:32:59.778607 | 2026-03-09 18:32:59.778771 | TASK [Slurp Zuul inventory test] 2026-03-09 18:33:00.204977 | controller -> localhost | ok 2026-03-09 18:33:00.224505 | 2026-03-09 18:33:00.224689 | TASK [Save zuul inventory] 2026-03-09 18:33:01.154366 | controller | changed 2026-03-09 18:33:01.169279 | 2026-03-09 18:33:01.169481 | TASK [Save zuul vars without the change_message] 2026-03-09 18:33:01.894732 | controller | changed 2026-03-09 18:33:01.942521 | 2026-03-09 18:33:01.942696 | PLAY RECAP 2026-03-09 18:33:01.942767 | controller | ok: 4 changed: 3 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-09 18:33:01.942816 | 2026-03-09 18:33:02.064396 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/dump_zuul_data.yml@main] 2026-03-09 18:33:02.073601 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-deps.yaml@main] 2026-03-09 18:33:02.742356 | 2026-03-09 18:33:02.742498 | PLAY [Run ci/playbooks/kuttl/deploy-deps.yml] 2026-03-09 18:33:02.766709 | 2026-03-09 18:33:02.766846 | TASK [Run kuttl deploy-deps playbook] 2026-03-09 18:33:04.100456 | controller | 2026-03-09 18:33:05.239044 | controller | PLAY [Bootstrap playbook] ****************************************************** 2026-03-09 18:33:05.239106 | controller | 2026-03-09 18:33:05.239112 | controller | TASK [Gathering Facts ] ******************************************************** 2026-03-09 18:33:05.239117 | controller | Monday 09 March 2026 18:33:04 +0000 (0:00:00.034) 0:00:00.034 ********** 2026-03-09 18:33:05.239122 | controller | Monday 09 March 2026 18:33:04 +0000 (0:00:00.033) 0:00:00.033 ********** 2026-03-09 18:33:05.239135 | controller | ok: [controller] 2026-03-09 18:33:05.271194 | controller | 2026-03-09 18:33:05.271261 | controller | TASK [cifmw_setup : Set custom cifmw PATH reusable fact cifmw_path={{ ansible_user_dir }}/.crc/bin:{{ ansible_user_dir }}/.crc/bin/oc:{{ ansible_user_dir }}/bin:{{ ansible_env.PATH }}, cacheable=True] *** 2026-03-09 18:33:05.271271 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:01.139) 0:00:01.173 ********** 2026-03-09 18:33:05.271276 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:01.139) 0:00:01.172 ********** 2026-03-09 18:33:05.271290 | controller | ok: [controller] 2026-03-09 18:33:05.352196 | controller | 2026-03-09 18:33:05.352263 | controller | TASK [cifmw_setup : Get customized parameters ci_framework_params={{ 2026-03-09 18:33:05.352271 | controller | hostvars[inventory_hostname] | 2026-03-09 18:33:05.352278 | controller | dict2items | 2026-03-09 18:33:05.352285 | controller | selectattr("key", "match", 2026-03-09 18:33:05.352291 | controller | "^(cifmw|pre|post)_(?!install_yamls|openshift_token|openshift_login|openshift_kubeconfig).*") | 2026-03-09 18:33:05.352298 | controller | list | items2dict 2026-03-09 18:33:05.352303 | controller | }}] *** 2026-03-09 18:33:05.352309 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.032) 0:00:01.206 ********** 2026-03-09 18:33:05.352315 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.032) 0:00:01.204 ********** 2026-03-09 18:33:05.352333 | controller | ok: [controller] 2026-03-09 18:33:05.724085 | controller | 2026-03-09 18:33:05.724139 | controller | TASK [install_ca : Ensure target directory exists path={{ cifmw_install_ca_trust_dir }}, state=directory, mode=0755] *** 2026-03-09 18:33:05.724146 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.079) 0:00:01.285 ********** 2026-03-09 18:33:05.724168 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.079) 0:00:01.283 ********** 2026-03-09 18:33:05.724179 | controller | ok: [controller] 2026-03-09 18:33:05.750198 | controller | 2026-03-09 18:33:05.750259 | controller | TASK [install_ca : Install internal CA from url url={{ cifmw_install_ca_url }}, dest={{ cifmw_install_ca_trust_dir }}, validate_certs={{ cifmw_install_ca_url_validate_certs | default(omit) }}, mode=0644] *** 2026-03-09 18:33:05.750266 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.373) 0:00:01.658 ********** 2026-03-09 18:33:05.750271 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.373) 0:00:01.657 ********** 2026-03-09 18:33:05.750294 | controller | skipping: [controller] 2026-03-09 18:33:05.776338 | controller | 2026-03-09 18:33:05.776373 | controller | TASK [install_ca : Install custom CA bundle from inline dest={{ cifmw_install_ca_trust_dir }}/cifmw_inline_ca_bundle.crt, content={{ cifmw_install_ca_bundle_inline }}, mode=0644] *** 2026-03-09 18:33:05.776380 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.026) 0:00:01.684 ********** 2026-03-09 18:33:05.776385 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.026) 0:00:01.683 ********** 2026-03-09 18:33:05.776396 | controller | skipping: [controller] 2026-03-09 18:33:05.802523 | controller | 2026-03-09 18:33:05.802556 | controller | TASK [install_ca : Install custom CA bundle from file dest={{ cifmw_install_ca_trust_dir }}/{{ cifmw_install_ca_bundle_src | basename }}, src={{ cifmw_install_ca_bundle_src }}, mode=0644] *** 2026-03-09 18:33:05.802565 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.024) 0:00:01.709 ********** 2026-03-09 18:33:05.802570 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.024) 0:00:01.708 ********** 2026-03-09 18:33:05.802580 | controller | skipping: [controller] 2026-03-09 18:33:05.835751 | controller | 2026-03-09 18:33:05.835806 | controller | TASK [install_ca : Update ca bundle _raw_params=update-ca-trust] *************** 2026-03-09 18:33:05.835820 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.026) 0:00:01.735 ********** 2026-03-09 18:33:05.835852 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.026) 0:00:01.734 ********** 2026-03-09 18:33:05.835866 | controller | skipping: [controller] 2026-03-09 18:33:06.933387 | controller | 2026-03-09 18:33:06.933440 | controller | TASK [repo_setup : Ensure directories are present path={{ cifmw_repo_setup_basedir }}/{{ item }}, state=directory, mode=0755] *** 2026-03-09 18:33:06.933447 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.035) 0:00:01.770 ********** 2026-03-09 18:33:06.933452 | controller | Monday 09 March 2026 18:33:05 +0000 (0:00:00.035) 0:00:01.769 ********** 2026-03-09 18:33:06.933467 | controller | changed: [controller] => (item=tmp) 2026-03-09 18:33:07.974403 | controller | changed: [controller] => (item=artifacts/repositories) 2026-03-09 18:33:07.974471 | controller | changed: [controller] => (item=venv/repo_setup) 2026-03-09 18:33:07.974479 | controller | 2026-03-09 18:33:07.974486 | controller | TASK [repo_setup : Make sure git-core package is installed name=git-core, state=present] *** 2026-03-09 18:33:07.974493 | controller | Monday 09 March 2026 18:33:06 +0000 (0:00:01.097) 0:00:02.868 ********** 2026-03-09 18:33:07.974499 | controller | Monday 09 March 2026 18:33:06 +0000 (0:00:01.097) 0:00:02.867 ********** 2026-03-09 18:33:07.974515 | controller | ok: [controller] 2026-03-09 18:33:08.846095 | controller | 2026-03-09 18:33:08.846182 | controller | TASK [repo_setup : Get repo-setup repository accept_hostkey=True, dest={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, repo={{ cifmw_repo_setup_src }}] *** 2026-03-09 18:33:08.846196 | controller | Monday 09 March 2026 18:33:07 +0000 (0:00:01.040) 0:00:03.909 ********** 2026-03-09 18:33:08.846203 | controller | Monday 09 March 2026 18:33:07 +0000 (0:00:01.040) 0:00:03.907 ********** 2026-03-09 18:33:08.846229 | controller | changed: [controller] 2026-03-09 18:33:16.780675 | controller | 2026-03-09 18:33:16.780732 | controller | TASK [repo_setup : Initialize python venv and install requirements virtualenv={{ cifmw_repo_setup_venv }}, requirements={{ cifmw_repo_setup_basedir }}/tmp/repo-setup/requirements.txt, virtualenv_command=python3 -m venv --system-site-packages --upgrade-deps] *** 2026-03-09 18:33:16.780743 | controller | Monday 09 March 2026 18:33:08 +0000 (0:00:00.871) 0:00:04.780 ********** 2026-03-09 18:33:16.780749 | controller | Monday 09 March 2026 18:33:08 +0000 (0:00:00.871) 0:00:04.779 ********** 2026-03-09 18:33:16.780762 | controller | changed: [controller] 2026-03-09 18:33:17.639765 | controller | 2026-03-09 18:33:17.639827 | controller | TASK [repo_setup : Install repo-setup package chdir={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, creates={{ cifmw_repo_setup_venv }}/bin/repo-setup, _raw_params={{ cifmw_repo_setup_venv }}/bin/python setup.py install] *** 2026-03-09 18:33:17.639853 | controller | Monday 09 March 2026 18:33:16 +0000 (0:00:07.934) 0:00:12.715 ********** 2026-03-09 18:33:17.639860 | controller | Monday 09 March 2026 18:33:16 +0000 (0:00:07.934) 0:00:12.714 ********** 2026-03-09 18:33:17.639871 | controller | changed: [controller] 2026-03-09 18:33:17.670622 | controller | 2026-03-09 18:33:17.670667 | controller | TASK [repo_setup : Set cifmw_repo_setup_dlrn_hash_tag from content provider cifmw_repo_setup_dlrn_hash_tag={{ content_provider_dlrn_md5_hash }}] *** 2026-03-09 18:33:17.670676 | controller | Monday 09 March 2026 18:33:17 +0000 (0:00:00.858) 0:00:13.574 ********** 2026-03-09 18:33:17.670682 | controller | Monday 09 March 2026 18:33:17 +0000 (0:00:00.859) 0:00:13.573 ********** 2026-03-09 18:33:17.670694 | controller | skipping: [controller] 2026-03-09 18:33:18.305025 | controller | 2026-03-09 18:33:18.305064 | controller | TASK [repo_setup : Run repo-setup _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup {{ cifmw_repo_setup_promotion }} {{ cifmw_repo_setup_additional_repos }} -d {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} -b {{ cifmw_repo_setup_branch }} --rdo-mirror {{ cifmw_repo_setup_rdo_mirror }} {% if cifmw_repo_setup_dlrn_hash_tag | length > 0 %} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif %} -o {{ cifmw_repo_setup_output }}] *** 2026-03-09 18:33:18.305073 | controller | Monday 09 March 2026 18:33:17 +0000 (0:00:00.031) 0:00:13.605 ********** 2026-03-09 18:33:18.305080 | controller | Monday 09 March 2026 18:33:17 +0000 (0:00:00.031) 0:00:13.604 ********** 2026-03-09 18:33:18.305097 | controller | changed: [controller] 2026-03-09 18:33:18.348129 | controller | 2026-03-09 18:33:18.348192 | controller | TASK [repo_setup : Get component repo url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/component/{{ cifmw_repo_setup_component_name }}/{{ cifmw_repo_setup_component_promotion_tag }}/delorean.repo, dest={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, mode=0644] *** 2026-03-09 18:33:18.348200 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.634) 0:00:14.239 ********** 2026-03-09 18:33:18.348205 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.634) 0:00:14.238 ********** 2026-03-09 18:33:18.348217 | controller | skipping: [controller] 2026-03-09 18:33:18.383463 | controller | 2026-03-09 18:33:18.383524 | controller | TASK [repo_setup : Rename component repo path={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, regexp=delorean-component-{{ cifmw_repo_setup_component_name }}, replace={{ cifmw_repo_setup_component_name }}-{{ cifmw_repo_setup_component_promotion_tag }}] *** 2026-03-09 18:33:18.383532 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.043) 0:00:14.282 ********** 2026-03-09 18:33:18.383538 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.043) 0:00:14.281 ********** 2026-03-09 18:33:18.383550 | controller | skipping: [controller] 2026-03-09 18:33:18.417458 | controller | 2026-03-09 18:33:18.417519 | controller | TASK [repo_setup : Disable component repo in current-podified dlrn repo path={{ cifmw_repo_setup_output }}/delorean.repo, section=delorean-component-{{ cifmw_repo_setup_component_name }}, option=enabled, value=0, mode=0644] *** 2026-03-09 18:33:18.417529 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.035) 0:00:14.318 ********** 2026-03-09 18:33:18.417536 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.035) 0:00:14.316 ********** 2026-03-09 18:33:18.417549 | controller | skipping: [controller] 2026-03-09 18:33:18.864104 | controller | 2026-03-09 18:33:18.864164 | controller | TASK [repo_setup : Run repo-setup-get-hash _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup-get-hash --dlrn-url {{ cifmw_repo_setup_dlrn_uri[:-1] }} --os-version {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} --release {{ cifmw_repo_setup_branch }} {% if cifmw_repo_setup_component_name | length > 0 -%} --component {{ cifmw_repo_setup_component_name }} --tag {{ cifmw_repo_setup_component_promotion_tag }} {% else -%} --tag {{cifmw_repo_setup_promotion }} {% endif -%} {% if (cifmw_repo_setup_dlrn_hash_tag | length > 0) and (cifmw_repo_setup_component_name | length <= 0) -%} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif -%} --json] *** 2026-03-09 18:33:18.864173 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.033) 0:00:14.352 ********** 2026-03-09 18:33:18.864181 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.033) 0:00:14.350 ********** 2026-03-09 18:33:18.864192 | controller | changed: [controller] 2026-03-09 18:33:19.678513 | controller | 2026-03-09 18:33:19.678549 | controller | TASK [repo_setup : Dump full hash in delorean.repo.md5 file content={{ _repo_setup_json['full_hash'] }} 2026-03-09 18:33:19.678556 | controller | , dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] *** 2026-03-09 18:33:19.678568 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.447) 0:00:14.799 ********** 2026-03-09 18:33:19.678573 | controller | Monday 09 March 2026 18:33:18 +0000 (0:00:00.446) 0:00:14.797 ********** 2026-03-09 18:33:19.678582 | controller | changed: [controller] 2026-03-09 18:33:19.696287 | controller | 2026-03-09 18:33:19.696323 | controller | TASK [repo_setup : Dump current-podified hash url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/current-podified/delorean.repo.md5, dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] *** 2026-03-09 18:33:19.696331 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.814) 0:00:15.613 ********** 2026-03-09 18:33:19.696342 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.814) 0:00:15.612 ********** 2026-03-09 18:33:19.696351 | controller | skipping: [controller] 2026-03-09 18:33:19.713347 | controller | 2026-03-09 18:33:19.713382 | controller | TASK [repo_setup : Slurp current podified hash src={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5] *** 2026-03-09 18:33:19.713388 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.631 ********** 2026-03-09 18:33:19.713394 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.629 ********** 2026-03-09 18:33:19.713403 | controller | skipping: [controller] 2026-03-09 18:33:19.730443 | controller | 2026-03-09 18:33:19.730477 | controller | TASK [repo_setup : Update the value of full_hash _repo_setup_json={{ _repo_setup_json | combine({'full_hash': _hash}, recursive=true) }}] *** 2026-03-09 18:33:19.730484 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.648 ********** 2026-03-09 18:33:19.730490 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.646 ********** 2026-03-09 18:33:19.730498 | controller | skipping: [controller] 2026-03-09 18:33:19.757999 | controller | 2026-03-09 18:33:19.758032 | controller | TASK [repo_setup : Export hashes facts for further use cifmw_repo_setup_full_hash={{ _repo_setup_json['full_hash'] }}, cifmw_repo_setup_commit_hash={{ _repo_setup_json['commit_hash'] }}, cifmw_repo_setup_distro_hash={{ _repo_setup_json['distro_hash'] }}, cifmw_repo_setup_extended_hash={{ _repo_setup_json['extended_hash'] }}, cifmw_repo_setup_dlrn_api_url={{ _repo_setup_json['dlrn_api_url'] }}, cifmw_repo_setup_dlrn_url={{ _repo_setup_json['dlrn_url'] }}, cifmw_repo_setup_release={{ _repo_setup_json['release'] }}, cacheable=True] *** 2026-03-09 18:33:19.758040 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.665 ********** 2026-03-09 18:33:19.758046 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.664 ********** 2026-03-09 18:33:19.758055 | controller | ok: [controller] 2026-03-09 18:33:19.774390 | controller | 2026-03-09 18:33:19.774431 | controller | TASK [repo_setup : Create download directory path={{ cifmw_repo_setup_rhos_release_path }}, state=directory, mode=0755] *** 2026-03-09 18:33:19.774437 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.027) 0:00:15.693 ********** 2026-03-09 18:33:19.774442 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.027) 0:00:15.691 ********** 2026-03-09 18:33:19.774451 | controller | skipping: [controller] 2026-03-09 18:33:19.791917 | controller | 2026-03-09 18:33:19.791956 | controller | TASK [repo_setup : Print the URL to request msg={{ cifmw_repo_setup_rhos_release_rpm }}] *** 2026-03-09 18:33:19.791964 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.016) 0:00:15.709 ********** 2026-03-09 18:33:19.791970 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.016) 0:00:15.707 ********** 2026-03-09 18:33:19.791981 | controller | skipping: [controller] 2026-03-09 18:33:19.810977 | controller | 2026-03-09 18:33:19.811009 | controller | TASK [Download the RPM name=krb_request] *************************************** 2026-03-09 18:33:19.811016 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.726 ********** 2026-03-09 18:33:19.811021 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.725 ********** 2026-03-09 18:33:19.811030 | controller | skipping: [controller] 2026-03-09 18:33:19.826135 | controller | 2026-03-09 18:33:19.826191 | controller | TASK [repo_setup : Install RHOS Release tool name={{ 2026-03-09 18:33:19.826199 | controller | cifmw_repo_setup_rhos_release_rpm 2026-03-09 18:33:19.826204 | controller | if cifmw_repo_setup_rhos_release_rpm is not url 2026-03-09 18:33:19.826208 | controller | else cifmw_krb_request_out.path 2026-03-09 18:33:19.826213 | controller | }}, state=present, disable_gpg_check={{ cifmw_repo_setup_rhos_release_gpg_check | bool }}] *** 2026-03-09 18:33:19.826218 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.019) 0:00:15.746 ********** 2026-03-09 18:33:19.826223 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.019) 0:00:15.744 ********** 2026-03-09 18:33:19.826232 | controller | skipping: [controller] 2026-03-09 18:33:19.840227 | controller | 2026-03-09 18:33:19.840263 | controller | TASK [repo_setup : Get rhos-release tool version _raw_params=rhos-release --version] *** 2026-03-09 18:33:19.840277 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.015) 0:00:15.761 ********** 2026-03-09 18:33:19.840282 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.015) 0:00:15.759 ********** 2026-03-09 18:33:19.840290 | controller | skipping: [controller] 2026-03-09 18:33:19.856286 | controller | 2026-03-09 18:33:19.856317 | controller | TASK [repo_setup : Print rhos-release tool version msg={{ rr_version.stdout }}] *** 2026-03-09 18:33:19.856328 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.014) 0:00:15.775 ********** 2026-03-09 18:33:19.856334 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.014) 0:00:15.773 ********** 2026-03-09 18:33:19.856342 | controller | skipping: [controller] 2026-03-09 18:33:19.870970 | controller | 2026-03-09 18:33:19.871002 | controller | TASK [repo_setup : Generate repos using rhos-release {{ cifmw_repo_setup_rhos_release_args }} _raw_params=rhos-release {{ cifmw_repo_setup_rhos_release_args }} \ 2026-03-09 18:33:19.871008 | controller | -t {{ cifmw_repo_setup_output }}] *** 2026-03-09 18:33:19.871014 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.015) 0:00:15.791 ********** 2026-03-09 18:33:19.871019 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.015) 0:00:15.789 ********** 2026-03-09 18:33:19.871027 | controller | skipping: [controller] 2026-03-09 18:33:20.079697 | controller | 2026-03-09 18:33:20.079733 | controller | TASK [repo_setup : Check for /etc/ci/mirror_info.sh path=/etc/ci/mirror_info.sh] *** 2026-03-09 18:33:20.079740 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.014) 0:00:15.805 ********** 2026-03-09 18:33:20.079745 | controller | Monday 09 March 2026 18:33:19 +0000 (0:00:00.014) 0:00:15.804 ********** 2026-03-09 18:33:20.079755 | controller | ok: [controller] 2026-03-09 18:33:20.294545 | controller | 2026-03-09 18:33:20.294593 | controller | TASK [repo_setup : Use RDO proxy mirrors chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail 2026-03-09 18:33:20.294600 | controller | source /etc/ci/mirror_info.sh 2026-03-09 18:33:20.294605 | controller | sed -i -e "s|https://trunk.rdoproject.org|$NODEPOOL_RDO_PROXY|g" *.repo 2026-03-09 18:33:20.294610 | controller | ] *** 2026-03-09 18:33:20.294615 | controller | Monday 09 March 2026 18:33:20 +0000 (0:00:00.208) 0:00:16.014 ********** 2026-03-09 18:33:20.294619 | controller | Monday 09 March 2026 18:33:20 +0000 (0:00:00.208) 0:00:16.013 ********** 2026-03-09 18:33:20.294631 | controller | changed: [controller] 2026-03-09 18:33:20.554331 | controller | 2026-03-09 18:33:20.554382 | controller | TASK [repo_setup : Use RDO CentOS mirrors (remove CentOS 10 conditional when Nodepool mirrors exist) chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail 2026-03-09 18:33:20.554390 | controller | source /etc/ci/mirror_info.sh 2026-03-09 18:33:20.554395 | controller | sed -i -e "s|http://mirror.stream.centos.org|$NODEPOOL_CENTOS_MIRROR|g" *.repo 2026-03-09 18:33:20.554401 | controller | ] *** 2026-03-09 18:33:20.554409 | controller | Monday 09 March 2026 18:33:20 +0000 (0:00:00.214) 0:00:16.229 ********** 2026-03-09 18:33:20.554413 | controller | Monday 09 March 2026 18:33:20 +0000 (0:00:00.214) 0:00:16.227 ********** 2026-03-09 18:33:20.554424 | controller | changed: [controller] 2026-03-09 18:33:21.121197 | controller | 2026-03-09 18:33:21.121233 | controller | TASK [repo_setup : Check for gating.repo file on content provider url=http://{{ content_provider_registry_ip }}:8766/gating.repo] *** 2026-03-09 18:33:21.121243 | controller | Monday 09 March 2026 18:33:20 +0000 (0:00:00.259) 0:00:16.489 ********** 2026-03-09 18:33:21.121250 | controller | Monday 09 March 2026 18:33:20 +0000 (0:00:00.259) 0:00:16.487 ********** 2026-03-09 18:33:21.121261 | controller | ok: [controller] 2026-03-09 18:33:21.692917 | controller | 2026-03-09 18:33:21.692993 | controller | TASK [repo_setup : Populate gating repo from content provider ip content=[gating-repo] 2026-03-09 18:33:21.693001 | controller | baseurl=http://{{ content_provider_registry_ip }}:8766/ 2026-03-09 18:33:21.693006 | controller | enabled=1 2026-03-09 18:33:21.693012 | controller | gpgcheck=0 2026-03-09 18:33:21.693019 | controller | priority=1 2026-03-09 18:33:21.693024 | controller | , dest={{ cifmw_repo_setup_output }}/gating.repo, mode=0644] *** 2026-03-09 18:33:21.693028 | controller | Monday 09 March 2026 18:33:21 +0000 (0:00:00.566) 0:00:17.056 ********** 2026-03-09 18:33:21.693033 | controller | Monday 09 March 2026 18:33:21 +0000 (0:00:00.566) 0:00:17.054 ********** 2026-03-09 18:33:21.693048 | controller | changed: [controller] 2026-03-09 18:33:21.921211 | controller | 2026-03-09 18:33:21.921281 | controller | TASK [repo_setup : Check for DLRN repo at the destination path={{ cifmw_repo_setup_output }}/delorean.repo] *** 2026-03-09 18:33:21.921288 | controller | Monday 09 March 2026 18:33:21 +0000 (0:00:00.571) 0:00:17.627 ********** 2026-03-09 18:33:21.921293 | controller | Monday 09 March 2026 18:33:21 +0000 (0:00:00.571) 0:00:17.626 ********** 2026-03-09 18:33:21.921308 | controller | ok: [controller] 2026-03-09 18:33:22.254531 | controller | 2026-03-09 18:33:22.254590 | controller | TASK [repo_setup : Lower the priority of DLRN repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/delorean.repo, regexp=priority=1, replace=priority=20] *** 2026-03-09 18:33:22.254597 | controller | Monday 09 March 2026 18:33:21 +0000 (0:00:00.228) 0:00:17.856 ********** 2026-03-09 18:33:22.254602 | controller | Monday 09 March 2026 18:33:21 +0000 (0:00:00.228) 0:00:17.854 ********** 2026-03-09 18:33:22.254615 | controller | changed: [controller] 2026-03-09 18:33:22.286415 | controller | 2026-03-09 18:33:22.286469 | controller | TASK [repo_setup : Check for DLRN component repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}] *** 2026-03-09 18:33:22.286476 | controller | Monday 09 March 2026 18:33:22 +0000 (0:00:00.333) 0:00:18.189 ********** 2026-03-09 18:33:22.286481 | controller | Monday 09 March 2026 18:33:22 +0000 (0:00:00.333) 0:00:18.187 ********** 2026-03-09 18:33:22.286493 | controller | skipping: [controller] 2026-03-09 18:33:22.316945 | controller | 2026-03-09 18:33:22.316998 | controller | TASK [repo_setup : Lower the priority of component repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}, regexp=priority=1, replace=priority=2] *** 2026-03-09 18:33:22.317004 | controller | Monday 09 March 2026 18:33:22 +0000 (0:00:00.031) 0:00:18.221 ********** 2026-03-09 18:33:22.317009 | controller | Monday 09 March 2026 18:33:22 +0000 (0:00:00.031) 0:00:18.219 ********** 2026-03-09 18:33:22.317021 | controller | skipping: [controller] 2026-03-09 18:33:22.707085 | controller | 2026-03-09 18:33:22.707118 | controller | TASK [repo_setup : Find existing repos from /etc/yum.repos.d directory paths=/etc/yum.repos.d/, patterns=*.repo, recurse=False] *** 2026-03-09 18:33:22.707126 | controller | Monday 09 March 2026 18:33:22 +0000 (0:00:00.030) 0:00:18.251 ********** 2026-03-09 18:33:22.707131 | controller | Monday 09 March 2026 18:33:22 +0000 (0:00:00.030) 0:00:18.250 ********** 2026-03-09 18:33:22.707140 | controller | ok: [controller] 2026-03-09 18:33:23.166827 | controller | 2026-03-09 18:33:23.166868 | controller | TASK [repo_setup : Remove existing repos from /etc/yum.repos.d directory path={{ item }}, state=absent] *** 2026-03-09 18:33:23.166875 | controller | Monday 09 March 2026 18:33:22 +0000 (0:00:00.390) 0:00:18.642 ********** 2026-03-09 18:33:23.166881 | controller | Monday 09 March 2026 18:33:22 +0000 (0:00:00.390) 0:00:18.640 ********** 2026-03-09 18:33:23.166893 | controller | changed: [controller] => (item=/etc/yum.repos.d/centos-addons.repo) 2026-03-09 18:33:23.586446 | controller | changed: [controller] => (item=/etc/yum.repos.d/centos.repo) 2026-03-09 18:33:23.586486 | controller | 2026-03-09 18:33:23.586493 | controller | TASK [repo_setup : Cleanup existing metadata _raw_params=dnf clean metadata] *** 2026-03-09 18:33:23.586498 | controller | Monday 09 March 2026 18:33:23 +0000 (0:00:00.459) 0:00:19.101 ********** 2026-03-09 18:33:23.586503 | controller | Monday 09 March 2026 18:33:23 +0000 (0:00:00.459) 0:00:19.100 ********** 2026-03-09 18:33:23.586520 | controller | changed: [controller] 2026-03-09 18:33:23.936756 | controller | 2026-03-09 18:33:23.936846 | controller | TASK [repo_setup : Copy generated repos to /etc/yum.repos.d directory mode=0755, remote_src=True, src={{ cifmw_repo_setup_output }}/, dest=/etc/yum.repos.d] *** 2026-03-09 18:33:23.936865 | controller | Monday 09 March 2026 18:33:23 +0000 (0:00:00.419) 0:00:19.521 ********** 2026-03-09 18:33:23.936881 | controller | Monday 09 March 2026 18:33:23 +0000 (0:00:00.419) 0:00:19.519 ********** 2026-03-09 18:33:23.936912 | controller | changed: [controller] 2026-03-09 18:33:23.991060 | controller | 2026-03-09 18:33:23.991117 | controller | TASK [ci_setup : Gather variables for each operating system _raw_params={{ item }}] *** 2026-03-09 18:33:23.991135 | controller | Monday 09 March 2026 18:33:23 +0000 (0:00:00.350) 0:00:19.871 ********** 2026-03-09 18:33:23.991142 | controller | Monday 09 March 2026 18:33:23 +0000 (0:00:00.350) 0:00:19.870 ********** 2026-03-09 18:33:23.991186 | controller | ok: [controller] => (item=/home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/ci_setup/vars/redhat.yml) 2026-03-09 18:33:24.013993 | controller | 2026-03-09 18:33:24.014055 | controller | TASK [ci_setup : List packages to install var=cifmw_ci_setup_packages] ********* 2026-03-09 18:33:24.014064 | controller | Monday 09 March 2026 18:33:23 +0000 (0:00:00.054) 0:00:19.926 ********** 2026-03-09 18:33:24.014070 | controller | Monday 09 March 2026 18:33:23 +0000 (0:00:00.054) 0:00:19.924 ********** 2026-03-09 18:33:24.014085 | controller | ok: [controller] => 2026-03-09 18:33:54.705942 | controller | cifmw_ci_setup_packages: 2026-03-09 18:33:54.706034 | controller | - bash-completion 2026-03-09 18:33:54.706042 | controller | - ca-certificates 2026-03-09 18:33:54.706046 | controller | - git-core 2026-03-09 18:33:54.706059 | controller | - make 2026-03-09 18:33:54.706063 | controller | - tar 2026-03-09 18:33:54.706068 | controller | - tmux 2026-03-09 18:33:54.706072 | controller | - python3-pip 2026-03-09 18:33:54.706077 | controller | 2026-03-09 18:33:54.706083 | controller | TASK [ci_setup : Install needed packages name={{ cifmw_ci_setup_packages }}, state=latest] *** 2026-03-09 18:33:54.706089 | controller | Monday 09 March 2026 18:33:24 +0000 (0:00:00.022) 0:00:19.948 ********** 2026-03-09 18:33:54.706094 | controller | Monday 09 March 2026 18:33:24 +0000 (0:00:00.022) 0:00:19.947 ********** 2026-03-09 18:33:54.706109 | controller | changed: [controller] 2026-03-09 18:33:54.954169 | controller | 2026-03-09 18:33:54.954204 | controller | TASK [ci_setup : Gather version of openshift client _raw_params=oc version --client -o yaml] *** 2026-03-09 18:33:54.954211 | controller | Monday 09 March 2026 18:33:54 +0000 (0:00:30.690) 0:00:50.639 ********** 2026-03-09 18:33:54.954216 | controller | Monday 09 March 2026 18:33:54 +0000 (0:00:30.690) 0:00:50.638 ********** 2026-03-09 18:33:54.954225 | controller | ok: [controller] 2026-03-09 18:33:55.157980 | controller | 2026-03-09 18:33:55.158036 | controller | TASK [ci_setup : Ensure openshift client install path is present path={{ cifmw_ci_setup_oc_install_path }}, state=directory, mode=0755] *** 2026-03-09 18:33:55.158043 | controller | Monday 09 March 2026 18:33:54 +0000 (0:00:00.249) 0:00:50.888 ********** 2026-03-09 18:33:55.158048 | controller | Monday 09 March 2026 18:33:54 +0000 (0:00:00.249) 0:00:50.887 ********** 2026-03-09 18:33:55.158060 | controller | changed: [controller] 2026-03-09 18:34:00.854569 | controller | 2026-03-09 18:34:00.854644 | controller | TASK [ci_setup : Install openshift client src={{ cifmw_ci_setup_openshift_client_download_uri }}/{{ cifmw_ci_setup_openshift_client_version }}/openshift-client-linux.tar.gz, dest={{ cifmw_ci_setup_oc_install_path }}, remote_src=True, mode=0755, creates={{ cifmw_ci_setup_oc_install_path }}/oc] *** 2026-03-09 18:34:00.854663 | controller | Monday 09 March 2026 18:33:55 +0000 (0:00:00.204) 0:00:51.092 ********** 2026-03-09 18:34:00.854671 | controller | Monday 09 March 2026 18:33:55 +0000 (0:00:00.204) 0:00:51.091 ********** 2026-03-09 18:34:00.854686 | controller | changed: [controller] 2026-03-09 18:34:00.884889 | controller | 2026-03-09 18:34:00.884944 | controller | TASK [ci_setup : Add the OC path to cifmw_path if needed cifmw_path={{ cifmw_ci_setup_oc_install_path }}:{{ ansible_env.PATH }}, cacheable=True] *** 2026-03-09 18:34:00.884952 | controller | Monday 09 March 2026 18:34:00 +0000 (0:00:05.696) 0:00:56.788 ********** 2026-03-09 18:34:00.884958 | controller | Monday 09 March 2026 18:34:00 +0000 (0:00:05.696) 0:00:56.787 ********** 2026-03-09 18:34:00.884970 | controller | skipping: [controller] 2026-03-09 18:34:01.221659 | controller | 2026-03-09 18:34:01.221783 | controller | TASK [ci_setup : Create completion file] *************************************** 2026-03-09 18:34:01.221794 | controller | Monday 09 March 2026 18:34:00 +0000 (0:00:00.030) 0:00:56.819 ********** 2026-03-09 18:34:01.221801 | controller | Monday 09 March 2026 18:34:00 +0000 (0:00:00.030) 0:00:56.818 ********** 2026-03-09 18:34:01.221825 | controller | changed: [controller] 2026-03-09 18:34:01.594507 | controller | 2026-03-09 18:34:01.594636 | controller | TASK [ci_setup : Source completion from within .bashrc create=True, mode=0644, path={{ ansible_user_dir }}/.bashrc, block=if [ -f ~/.oc_completion ]; then 2026-03-09 18:34:01.594674 | controller | source ~/.oc_completion 2026-03-09 18:34:01.594684 | controller | fi] *** 2026-03-09 18:34:01.594692 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.336) 0:00:57.156 ********** 2026-03-09 18:34:01.594701 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.336) 0:00:57.155 ********** 2026-03-09 18:34:01.594729 | controller | changed: [controller] 2026-03-09 18:34:01.619106 | controller | 2026-03-09 18:34:01.619184 | controller | TASK [ci_setup : Check rhsm status _raw_params=subscription-manager status] **** 2026-03-09 18:34:01.619193 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.372) 0:00:57.529 ********** 2026-03-09 18:34:01.619198 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.372) 0:00:57.527 ********** 2026-03-09 18:34:01.619211 | controller | skipping: [controller] 2026-03-09 18:34:01.646039 | controller | 2026-03-09 18:34:01.646125 | controller | TASK [ci_setup : Gather the repos to be enabled _repos={{ 2026-03-09 18:34:01.646133 | controller | cifmw_ci_setup_rhel_rhsm_default_repos + 2026-03-09 18:34:01.646139 | controller | (cifmw_ci_setup_rhel_rhsm_extra_repos | default([])) 2026-03-09 18:34:01.646144 | controller | }}] *** 2026-03-09 18:34:01.646171 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.024) 0:00:57.553 ********** 2026-03-09 18:34:01.646177 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.024) 0:00:57.552 ********** 2026-03-09 18:34:01.646195 | controller | skipping: [controller] 2026-03-09 18:34:01.667941 | controller | 2026-03-09 18:34:01.668040 | controller | TASK [ci_setup : Enabling the required repositories. name={{ item }}, state={{ rhsm_repo_state | default('enabled') }}] *** 2026-03-09 18:34:01.668055 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.026) 0:00:57.580 ********** 2026-03-09 18:34:01.668065 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.026) 0:00:57.579 ********** 2026-03-09 18:34:01.668120 | controller | skipping: [controller] 2026-03-09 18:34:01.691595 | controller | 2026-03-09 18:34:01.691645 | controller | TASK [ci_setup : Get current /etc/redhat-release _raw_params=cat /etc/redhat-release] *** 2026-03-09 18:34:01.691653 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.022) 0:00:57.602 ********** 2026-03-09 18:34:01.691659 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.022) 0:00:57.601 ********** 2026-03-09 18:34:01.691669 | controller | skipping: [controller] 2026-03-09 18:34:01.710066 | controller | 2026-03-09 18:34:01.710106 | controller | TASK [ci_setup : Print current /etc/redhat-release msg={{ _current_rh_release.stdout }}] *** 2026-03-09 18:34:01.710113 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.023) 0:00:57.626 ********** 2026-03-09 18:34:01.710118 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.023) 0:00:57.624 ********** 2026-03-09 18:34:01.710128 | controller | skipping: [controller] 2026-03-09 18:34:01.733198 | controller | 2026-03-09 18:34:01.733237 | controller | TASK [ci_setup : Ensure the repos are enabled in the system using yum name={{ item.name }}, baseurl={{ item.baseurl }}, description={{ item.description | default(item.name) }}, gpgcheck={{ item.gpgcheck | default(false) }}, enabled=True, state={{ yum_repo_state | default('present') }}] *** 2026-03-09 18:34:01.733255 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.018) 0:00:57.644 ********** 2026-03-09 18:34:01.733269 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.018) 0:00:57.643 ********** 2026-03-09 18:34:01.733290 | controller | skipping: [controller] 2026-03-09 18:34:02.905412 | controller | 2026-03-09 18:34:02.905491 | controller | TASK [ci_setup : Manage directories path={{ item }}, state={{ directory_state }}, mode=0755, owner={{ ansible_user_id }}, group={{ ansible_user_id }}] *** 2026-03-09 18:34:02.905498 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.023) 0:00:57.667 ********** 2026-03-09 18:34:02.905503 | controller | Monday 09 March 2026 18:34:01 +0000 (0:00:00.023) 0:00:57.666 ********** 2026-03-09 18:34:02.905519 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/manifests/openstack/cr) 2026-03-09 18:34:03.102708 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/logs) 2026-03-09 18:34:03.102776 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/tmp) 2026-03-09 18:34:03.102793 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/volumes) 2026-03-09 18:34:03.102798 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-09 18:34:03.102803 | controller | 2026-03-09 18:34:03.102807 | controller | TASK [Prepare install_yamls make targets name=install_yamls, apply={'tags': ['bootstrap']}] *** 2026-03-09 18:34:03.102812 | controller | Monday 09 March 2026 18:34:02 +0000 (0:00:01.171) 0:00:58.839 ********** 2026-03-09 18:34:03.102817 | controller | Monday 09 March 2026 18:34:02 +0000 (0:00:01.171) 0:00:58.838 ********** 2026-03-09 18:34:03.102831 | controller | 2026-03-09 18:34:03.753655 | controller | TASK [install_yamls : Ensure directories exist path={{ item }}, state=directory, mode=0755] *** 2026-03-09 18:34:03.753689 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.197) 0:00:59.037 ********** 2026-03-09 18:34:03.753702 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.197) 0:00:59.036 ********** 2026-03-09 18:34:03.753712 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts) 2026-03-09 18:34:03.788636 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/roles/install_yamls_makes/tasks) 2026-03-09 18:34:03.788672 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-09 18:34:03.788678 | controller | 2026-03-09 18:34:03.788683 | controller | TASK [Create variables with local repos based on Zuul items name=install_yamls, tasks_from=zuul_set_operators_repo.yml] *** 2026-03-09 18:34:03.788688 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.650) 0:00:59.688 ********** 2026-03-09 18:34:03.788693 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.650) 0:00:59.687 ********** 2026-03-09 18:34:03.788701 | controller | 2026-03-09 18:34:03.875349 | controller | TASK [install_yamls : Set fact with local repos based on Zuul items cifmw_install_yamls_operators_repo={{ cifmw_install_yamls_operators_repo | default({}) | combine(_repo_operator_info | items2dict) }}] *** 2026-03-09 18:34:03.875389 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.035) 0:00:59.723 ********** 2026-03-09 18:34:03.875399 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.035) 0:00:59.722 ********** 2026-03-09 18:34:03.875412 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '979276', 'change_url': 'https://review.opendev.org/c/openstack/watcher/+/979276', 'commit_id': '64164197e0a8ae9e5016567724b13e7a66130ae7', 'patchset': '3', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher', 'name': 'openstack/watcher', 'short_name': 'watcher', 'src_dir': 'src/opendev.org/openstack/watcher'}, 'topic': None}) 2026-03-09 18:34:03.964363 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '979643', 'change_url': 'https://review.opendev.org/c/openstack/watcher/+/979643', 'commit_id': '29e4137808ab2295c733b36b9851bdace3dec60a', 'patchset': '3', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher', 'name': 'openstack/watcher', 'short_name': 'watcher', 'src_dir': 'src/opendev.org/openstack/watcher'}, 'topic': None}) 2026-03-09 18:34:03.964399 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '958644', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/958644', 'commit_id': '5931d8d6e69fb6fa67320bbc508b3744b12bed85', 'patchset': '32', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03.964407 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '962702', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/962702', 'commit_id': 'b0c5f8ac8fc6c508dc18eab2816d38f4c959083b', 'patchset': '25', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03.964418 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '963559', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559', 'commit_id': '6fedf196e61ec8ee8c9655e8fa463f714bd0ce13', 'patchset': '16', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03.964423 | controller | ok: [controller] => (item={'branch': 'main', 'change': '351', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/351', 'commit_id': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'patchset': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None}) 2026-03-09 18:34:03.964428 | controller | 2026-03-09 18:34:03.964434 | controller | TASK [install_yamls : Print helpful data for debugging msg=_repo_operator_name: {{ _repo_operator_name }} 2026-03-09 18:34:03.964439 | controller | _repo_operator_info: {{ _repo_operator_info }} 2026-03-09 18:34:03.964444 | controller | cifmw_install_yamls_operators_repo: {{ cifmw_install_yamls_operators_repo }} 2026-03-09 18:34:03.964449 | controller | ] *** 2026-03-09 18:34:03.964454 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.086) 0:00:59.810 ********** 2026-03-09 18:34:03.964458 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.086) 0:00:59.808 ********** 2026-03-09 18:34:03.964467 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '979276', 'change_url': 'https://review.opendev.org/c/openstack/watcher/+/979276', 'commit_id': '64164197e0a8ae9e5016567724b13e7a66130ae7', 'patchset': '3', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher', 'name': 'openstack/watcher', 'short_name': 'watcher', 'src_dir': 'src/opendev.org/openstack/watcher'}, 'topic': None}) 2026-03-09 18:34:04.003075 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '979643', 'change_url': 'https://review.opendev.org/c/openstack/watcher/+/979643', 'commit_id': '29e4137808ab2295c733b36b9851bdace3dec60a', 'patchset': '3', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher', 'name': 'openstack/watcher', 'short_name': 'watcher', 'src_dir': 'src/opendev.org/openstack/watcher'}, 'topic': None}) 2026-03-09 18:34:04.003112 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '958644', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/958644', 'commit_id': '5931d8d6e69fb6fa67320bbc508b3744b12bed85', 'patchset': '32', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:04.003124 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '962702', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/962702', 'commit_id': 'b0c5f8ac8fc6c508dc18eab2816d38f4c959083b', 'patchset': '25', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:04.003129 | controller | skipping: [controller] => (item={'branch': 'master', 'change': '963559', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559', 'commit_id': '6fedf196e61ec8ee8c9655e8fa463f714bd0ce13', 'patchset': '16', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:04.003140 | controller | ok: [controller] => (item={'branch': 'main', 'change': '351', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/351', 'commit_id': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'patchset': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None}) => 2026-03-09 18:34:04.003145 | controller | msg: | 2026-03-09 18:34:04.003172 | controller | _repo_operator_name: watcher 2026-03-09 18:34:04.003178 | controller | _repo_operator_info: [{'key': 'WATCHER_REPO', 'value': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}, {'key': 'WATCHER_BRANCH', 'value': ''}] 2026-03-09 18:34:04.003184 | controller | cifmw_install_yamls_operators_repo: {'WATCHER_REPO': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator', 'WATCHER_BRANCH': ''} 2026-03-09 18:34:04.003189 | controller | 2026-03-09 18:34:04.003194 | controller | TASK [Customize install_yamls devsetup vars if needed name=install_yamls, tasks_from=customize_devsetup_vars.yml] *** 2026-03-09 18:34:04.003198 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.089) 0:00:59.899 ********** 2026-03-09 18:34:04.003203 | controller | Monday 09 March 2026 18:34:03 +0000 (0:00:00.089) 0:00:59.897 ********** 2026-03-09 18:34:04.003212 | controller | 2026-03-09 18:34:04.025530 | controller | TASK [install_yamls : Update opm_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^opm_version:, line=opm_version: {{ cifmw_install_yamls_opm_version }}, state=present] *** 2026-03-09 18:34:04.025591 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.038) 0:00:59.938 ********** 2026-03-09 18:34:04.025604 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.038) 0:00:59.936 ********** 2026-03-09 18:34:04.025627 | controller | skipping: [controller] 2026-03-09 18:34:04.048304 | controller | 2026-03-09 18:34:04.048361 | controller | TASK [install_yamls : Update sdk_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^sdk_version:, line=sdk_version: {{ cifmw_install_yamls_sdk_version }}, state=present] *** 2026-03-09 18:34:04.048368 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.022) 0:00:59.960 ********** 2026-03-09 18:34:04.048374 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.022) 0:00:59.959 ********** 2026-03-09 18:34:04.048394 | controller | skipping: [controller] 2026-03-09 18:34:04.070429 | controller | 2026-03-09 18:34:04.070485 | controller | TASK [install_yamls : Update go_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^go_version:, line=go_version: {{ cifmw_install_yamls_go_version }}, state=present] *** 2026-03-09 18:34:04.070492 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.022) 0:00:59.983 ********** 2026-03-09 18:34:04.070498 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.022) 0:00:59.981 ********** 2026-03-09 18:34:04.070510 | controller | skipping: [controller] 2026-03-09 18:34:04.099313 | controller | 2026-03-09 18:34:04.099386 | controller | TASK [install_yamls : Update kustomize_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^kustomize_version:, line=kustomize_version: {{ cifmw_install_yamls_kustomize_version }}, state=present] *** 2026-03-09 18:34:04.099396 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.021) 0:01:00.005 ********** 2026-03-09 18:34:04.099403 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.021) 0:01:00.003 ********** 2026-03-09 18:34:04.099441 | controller | skipping: [controller] 2026-03-09 18:34:04.194601 | controller | 2026-03-09 18:34:04.194668 | controller | TASK [install_yamls : Compute the cifmw_install_yamls_vars final value _install_yamls_override_vars={{ 2026-03-09 18:34:04.194675 | controller | _install_yamls_override_vars | default({}) | 2026-03-09 18:34:04.194680 | controller | combine(item, recursive=True) 2026-03-09 18:34:04.194685 | controller | }}] *** 2026-03-09 18:34:04.194690 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.028) 0:01:00.034 ********** 2026-03-09 18:34:04.194694 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.028) 0:01:00.032 ********** 2026-03-09 18:34:04.194708 | controller | ok: [controller] => (item={'BMO_SETUP': False}) 2026-03-09 18:34:04.238693 | controller | 2026-03-09 18:34:04.238769 | controller | TASK [install_yamls : Set environment override cifmw_install_yamls_environment fact cifmw_install_yamls_environment={{ 2026-03-09 18:34:04.238780 | controller | _install_yamls_override_vars.keys() | 2026-03-09 18:34:04.238788 | controller | map('upper') | 2026-03-09 18:34:04.238794 | controller | zip(_install_yamls_override_vars.values()) | 2026-03-09 18:34:04.238801 | controller | items2dict(key_name=0, value_name=1) | 2026-03-09 18:34:04.238807 | controller | combine({ 2026-03-09 18:34:04.238813 | controller | 'OUT': cifmw_install_yamls_manifests_dir, 2026-03-09 18:34:04.238820 | controller | 'OUTPUT_DIR': cifmw_install_yamls_edpm_dir, 2026-03-09 18:34:04.238827 | controller | 'CHECKOUT_FROM_OPENSTACK_REF': cifmw_install_yamls_checkout_openstack_ref, 2026-03-09 18:34:04.238834 | controller | 'OPENSTACK_K8S_BRANCH': (zuul is defined and not zuul.branch |regex_search('master|antelope|rhos')) | ternary(zuul.branch, 'main') 2026-03-09 18:34:04.238841 | controller | }) | 2026-03-09 18:34:04.238846 | controller | combine(install_yamls_operators_repos) 2026-03-09 18:34:04.238852 | controller | }}, cacheable=True] *** 2026-03-09 18:34:04.238859 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.095) 0:01:00.129 ********** 2026-03-09 18:34:04.238865 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.095) 0:01:00.128 ********** 2026-03-09 18:34:04.238882 | controller | ok: [controller] 2026-03-09 18:34:04.821508 | controller | 2026-03-09 18:34:04.821546 | controller | TASK [install_yamls : Get environment structure base_path={{ cifmw_install_yamls_repo }}] *** 2026-03-09 18:34:04.821553 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.044) 0:01:00.173 ********** 2026-03-09 18:34:04.821558 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.044) 0:01:00.172 ********** 2026-03-09 18:34:04.821567 | controller | ok: [controller] 2026-03-09 18:34:05.034658 | controller | 2026-03-09 18:34:05.034692 | controller | TASK [install_yamls : Ensure Output directory exists path={{ cifmw_install_yamls_out_dir }}, state=directory, mode=0755] *** 2026-03-09 18:34:05.034699 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.582) 0:01:00.756 ********** 2026-03-09 18:34:05.034704 | controller | Monday 09 March 2026 18:34:04 +0000 (0:00:00.582) 0:01:00.755 ********** 2026-03-09 18:34:05.034713 | controller | ok: [controller] 2026-03-09 18:34:05.072372 | controller | 2026-03-09 18:34:05.072437 | controller | TASK [install_yamls : Ensure user cifmw_install_yamls_vars contains existing Makefile variables that=_cifmw_install_yamls_unmatched_vars | length == 0, msg=cifmw_install_yamls_vars contains a variable that is not defined in install_yamls Makefile nor cifmw_install_yamls_whitelisted_vars: {{ _cifmw_install_yamls_unmatched_vars | join(', ')}}, quiet=True] *** 2026-03-09 18:34:05.072456 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.213) 0:01:00.969 ********** 2026-03-09 18:34:05.072473 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.213) 0:01:00.968 ********** 2026-03-09 18:34:05.072498 | controller | ok: [controller] 2026-03-09 18:34:05.617244 | controller | 2026-03-09 18:34:05.617302 | controller | TASK [install_yamls : Generate /home/zuul/ci-framework-data/artifacts/install_yamls.sh dest={{ cifmw_install_yamls_out_dir }}/{{ cifmw_install_yamls_envfile }}, content={% for k,v in cifmw_install_yamls_environment.items() %} 2026-03-09 18:34:05.617309 | controller | export {{ k }}={{ v }} 2026-03-09 18:34:05.617314 | controller | {% endfor %}, mode=0644] *** 2026-03-09 18:34:05.617319 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.037) 0:01:01.007 ********** 2026-03-09 18:34:05.617325 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.037) 0:01:01.005 ********** 2026-03-09 18:34:05.617346 | controller | changed: [controller] 2026-03-09 18:34:05.648716 | controller | 2026-03-09 18:34:05.648779 | controller | TASK [install_yamls : Set install_yamls default values cifmw_install_yamls_defaults={{ get_makefiles_env_output.makefiles_values | combine(cifmw_install_yamls_environment) }}, cacheable=True] *** 2026-03-09 18:34:05.648787 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.544) 0:01:01.552 ********** 2026-03-09 18:34:05.648792 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.544) 0:01:01.550 ********** 2026-03-09 18:34:05.648806 | controller | ok: [controller] 2026-03-09 18:34:05.671349 | controller | 2026-03-09 18:34:05.671408 | controller | TASK [install_yamls : Show the env structure var=cifmw_install_yamls_environment] *** 2026-03-09 18:34:05.671416 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.031) 0:01:01.583 ********** 2026-03-09 18:34:05.671422 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.031) 0:01:01.582 ********** 2026-03-09 18:34:05.671438 | controller | ok: [controller] => 2026-03-09 18:34:05.698863 | controller | cifmw_install_yamls_environment: 2026-03-09 18:34:05.698922 | controller | BMO_SETUP: false 2026-03-09 18:34:05.698928 | controller | CHECKOUT_FROM_OPENSTACK_REF: 'true' 2026-03-09 18:34:05.698934 | controller | OPENSTACK_K8S_BRANCH: main 2026-03-09 18:34:05.698938 | controller | OUT: /home/zuul/ci-framework-data/artifacts/manifests 2026-03-09 18:34:05.698948 | controller | OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm 2026-03-09 18:34:05.698953 | controller | WATCHER_BRANCH: '' 2026-03-09 18:34:05.698958 | controller | WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator 2026-03-09 18:34:05.698964 | controller | 2026-03-09 18:34:05.698969 | controller | TASK [install_yamls : Show the env structure defaults var=cifmw_install_yamls_defaults] *** 2026-03-09 18:34:05.698974 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.022) 0:01:01.606 ********** 2026-03-09 18:34:05.698980 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.022) 0:01:01.604 ********** 2026-03-09 18:34:05.698992 | controller | ok: [controller] => 2026-03-09 18:34:05.698998 | controller | cifmw_install_yamls_defaults: 2026-03-09 18:34:05.699006 | controller | ADOPTED_EXTERNAL_NETWORK: 172.21.1.0/24 2026-03-09 18:34:05.699010 | controller | ADOPTED_INTERNALAPI_NETWORK: 172.17.1.0/24 2026-03-09 18:34:05.699015 | controller | ADOPTED_STORAGEMGMT_NETWORK: 172.20.1.0/24 2026-03-09 18:34:05.699019 | controller | ADOPTED_STORAGE_NETWORK: 172.18.1.0/24 2026-03-09 18:34:05.699024 | controller | ADOPTED_TENANT_NETWORK: 172.9.1.0/24 2026-03-09 18:34:05.699028 | controller | BARBICAN: config/samples/barbican_v1beta1_barbican.yaml 2026-03-09 18:34:05.699032 | controller | BARBICAN_BRANCH: main 2026-03-09 18:34:05.699037 | controller | BARBICAN_COMMIT_HASH: '' 2026-03-09 18:34:05.699042 | controller | BARBICAN_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/config/samples/barbican_v1beta1_barbican.yaml 2026-03-09 18:34:05.699047 | controller | BARBICAN_DEPL_IMG: unused 2026-03-09 18:34:05.699051 | controller | BARBICAN_IMG: quay.io/openstack-k8s-operators/barbican-operator-index:latest 2026-03-09 18:34:05.699056 | controller | BARBICAN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/kuttl-test.yaml 2026-03-09 18:34:05.699060 | controller | BARBICAN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/test/kuttl/tests 2026-03-09 18:34:05.699065 | controller | BARBICAN_KUTTL_NAMESPACE: barbican-kuttl-tests 2026-03-09 18:34:05.699069 | controller | BARBICAN_REPO: https://github.com/openstack-k8s-operators/barbican-operator.git 2026-03-09 18:34:05.699074 | controller | BARBICAN_SERVICE_ENABLED: 'true' 2026-03-09 18:34:05.699078 | controller | BARBICAN_SIMPLE_CRYPTO_ENCRYPTION_KEY: sEFmdFjDUqRM2VemYslV5yGNWjokioJXsg8Nrlc3drU= 2026-03-09 18:34:05.699082 | controller | BAREMETAL_BRANCH: main 2026-03-09 18:34:05.699087 | controller | BAREMETAL_COMMIT_HASH: '' 2026-03-09 18:34:05.699091 | controller | BAREMETAL_IMG: quay.io/openstack-k8s-operators/openstack-baremetal-operator-index:latest 2026-03-09 18:34:05.699096 | controller | BAREMETAL_OS_CONTAINER_IMG: '' 2026-03-09 18:34:05.699100 | controller | BAREMETAL_OS_IMG: '' 2026-03-09 18:34:05.699111 | controller | BAREMETAL_OS_IMG_TYPE: '' 2026-03-09 18:34:05.699116 | controller | BAREMETAL_REPO: https://github.com/openstack-k8s-operators/openstack-baremetal-operator.git 2026-03-09 18:34:05.699120 | controller | BAREMETAL_TIMEOUT: 20m 2026-03-09 18:34:05.699124 | controller | BASH_IMG: quay.io/openstack-k8s-operators/bash:latest 2026-03-09 18:34:05.699129 | controller | BGP_ASN: '64999' 2026-03-09 18:34:05.699133 | controller | BGP_LEAF_1: 100.65.4.1 2026-03-09 18:34:05.699137 | controller | BGP_LEAF_2: 100.64.4.1 2026-03-09 18:34:05.699142 | controller | BGP_OVN_ROUTING: 'false' 2026-03-09 18:34:05.699146 | controller | BGP_PEER_ASN: '64999' 2026-03-09 18:34:05.699174 | controller | BGP_SOURCE_IP: 172.30.4.2 2026-03-09 18:34:05.699179 | controller | BGP_SOURCE_IP6: f00d:f00d:f00d:f00d:f00d:f00d:f00d:42 2026-03-09 18:34:05.699183 | controller | BMAAS_BRIDGE_IPV4_PREFIX: 172.20.1.2/24 2026-03-09 18:34:05.699188 | controller | BMAAS_BRIDGE_IPV6_PREFIX: fd00:bbbb::2/64 2026-03-09 18:34:05.699192 | controller | BMAAS_INSTANCE_DISK_SIZE: '20' 2026-03-09 18:34:05.699196 | controller | BMAAS_INSTANCE_MEMORY: '4096' 2026-03-09 18:34:05.699201 | controller | BMAAS_INSTANCE_NAME_PREFIX: crc-bmaas 2026-03-09 18:34:05.699206 | controller | BMAAS_INSTANCE_NET_MODEL: virtio 2026-03-09 18:34:05.699210 | controller | BMAAS_INSTANCE_OS_VARIANT: centos-stream9 2026-03-09 18:34:05.699214 | controller | BMAAS_INSTANCE_VCPUS: '2' 2026-03-09 18:34:05.699219 | controller | BMAAS_INSTANCE_VIRT_TYPE: kvm 2026-03-09 18:34:05.699223 | controller | BMAAS_IPV4: 'true' 2026-03-09 18:34:05.699228 | controller | BMAAS_IPV6: 'false' 2026-03-09 18:34:05.699232 | controller | BMAAS_LIBVIRT_USER: sushyemu 2026-03-09 18:34:05.699236 | controller | BMAAS_METALLB_ADDRESS_POOL: 172.20.1.64/26 2026-03-09 18:34:05.699241 | controller | BMAAS_METALLB_POOL_NAME: baremetal 2026-03-09 18:34:05.699245 | controller | BMAAS_NETWORK_IPV4_PREFIX: 172.20.1.1/24 2026-03-09 18:34:05.699249 | controller | BMAAS_NETWORK_IPV6_PREFIX: fd00:bbbb::1/64 2026-03-09 18:34:05.699254 | controller | BMAAS_NETWORK_NAME: crc-bmaas 2026-03-09 18:34:05.699258 | controller | BMAAS_NODE_COUNT: '1' 2026-03-09 18:34:05.699262 | controller | BMAAS_OCP_INSTANCE_NAME: crc 2026-03-09 18:34:05.699267 | controller | BMAAS_REDFISH_PASSWORD: password 2026-03-09 18:34:05.699271 | controller | BMAAS_REDFISH_USERNAME: admin 2026-03-09 18:34:05.699275 | controller | BMAAS_ROUTE_LIBVIRT_NETWORKS: crc-bmaas,crc,default 2026-03-09 18:34:05.699280 | controller | BMAAS_SUSHY_EMULATOR_DRIVER: libvirt 2026-03-09 18:34:05.699284 | controller | BMAAS_SUSHY_EMULATOR_IMAGE: quay.io/metal3-io/sushy-tools:latest 2026-03-09 18:34:05.699288 | controller | BMAAS_SUSHY_EMULATOR_NAMESPACE: sushy-emulator 2026-03-09 18:34:05.699293 | controller | BMAAS_SUSHY_EMULATOR_OS_CLIENT_CONFIG_FILE: /etc/openstack/clouds.yaml 2026-03-09 18:34:05.699297 | controller | BMAAS_SUSHY_EMULATOR_OS_CLOUD: openstack 2026-03-09 18:34:05.699301 | controller | BMH_NAMESPACE: openstack 2026-03-09 18:34:05.699306 | controller | BMO_BRANCH: release-0.9 2026-03-09 18:34:05.699310 | controller | BMO_CLEANUP: 'true' 2026-03-09 18:34:05.699315 | controller | BMO_COMMIT_HASH: '' 2026-03-09 18:34:05.699319 | controller | BMO_IPA_BASEURI: https://tarballs.opendev.org/openstack/ironic-python-agent/dib 2026-03-09 18:34:05.699323 | controller | BMO_IPA_BRANCH: stable/2024.1 2026-03-09 18:34:05.699352 | controller | BMO_IPA_FLAVOR: centos9 2026-03-09 18:34:05.699357 | controller | BMO_IPA_INSECURE: 'false' 2026-03-09 18:34:05.699361 | controller | BMO_IRONIC_HOST: 192.168.122.10 2026-03-09 18:34:05.699365 | controller | BMO_PROVISIONING_INTERFACE: '' 2026-03-09 18:34:05.699370 | controller | BMO_REPO: https://github.com/metal3-io/baremetal-operator 2026-03-09 18:34:05.699374 | controller | BMO_SETUP: false 2026-03-09 18:34:05.699379 | controller | BMO_SETUP_ROUTE_REPLACE: 'true' 2026-03-09 18:34:05.699383 | controller | BM_CTLPLANE_INTERFACE: enp1s0 2026-03-09 18:34:05.699387 | controller | BM_INSTANCE_MEMORY: '8192' 2026-03-09 18:34:05.699392 | controller | BM_INSTANCE_NAME_PREFIX: edpm-compute-baremetal 2026-03-09 18:34:05.699396 | controller | BM_INSTANCE_NAME_SUFFIX: '0' 2026-03-09 18:34:05.699400 | controller | BM_NETWORK_NAME: default 2026-03-09 18:34:05.699405 | controller | BM_NODE_COUNT: '1' 2026-03-09 18:34:05.699409 | controller | BM_ROOT_PASSWORD: '' 2026-03-09 18:34:05.699414 | controller | BM_ROOT_PASSWORD_SECRET: '' 2026-03-09 18:34:05.699421 | controller | CEILOMETER_CENTRAL_DEPL_IMG: unused 2026-03-09 18:34:05.699426 | controller | CEILOMETER_NOTIFICATION_DEPL_IMG: unused 2026-03-09 18:34:05.699430 | controller | CEPH_BRANCH: release-1.15 2026-03-09 18:34:05.699435 | controller | CEPH_CLIENT: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/toolbox.yaml 2026-03-09 18:34:05.699443 | controller | CEPH_COMMON: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/common.yaml 2026-03-09 18:34:05.699448 | controller | CEPH_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/cluster-test.yaml 2026-03-09 18:34:05.699453 | controller | CEPH_CRDS: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/crds.yaml 2026-03-09 18:34:05.699457 | controller | CEPH_IMG: quay.io/ceph/demo:latest-squid 2026-03-09 18:34:05.699462 | controller | CEPH_OP: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/operator-openshift.yaml 2026-03-09 18:34:05.699466 | controller | CEPH_REPO: https://github.com/rook/rook.git 2026-03-09 18:34:05.699471 | controller | CERTMANAGER_TIMEOUT: 300s 2026-03-09 18:34:05.699475 | controller | CHECKOUT_FROM_OPENSTACK_REF: 'true' 2026-03-09 18:34:05.699479 | controller | CINDER: config/samples/cinder_v1beta1_cinder.yaml 2026-03-09 18:34:05.699484 | controller | CINDERAPI_DEPL_IMG: unused 2026-03-09 18:34:05.699488 | controller | CINDERBKP_DEPL_IMG: unused 2026-03-09 18:34:05.699492 | controller | CINDERSCH_DEPL_IMG: unused 2026-03-09 18:34:05.699497 | controller | CINDERVOL_DEPL_IMG: unused 2026-03-09 18:34:05.699501 | controller | CINDER_BRANCH: main 2026-03-09 18:34:05.699506 | controller | CINDER_COMMIT_HASH: '' 2026-03-09 18:34:05.699510 | controller | CINDER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/config/samples/cinder_v1beta1_cinder.yaml 2026-03-09 18:34:05.699515 | controller | CINDER_IMG: quay.io/openstack-k8s-operators/cinder-operator-index:latest 2026-03-09 18:34:05.699519 | controller | CINDER_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/kuttl-test.yaml 2026-03-09 18:34:05.699524 | controller | CINDER_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/test/kuttl/tests 2026-03-09 18:34:05.699528 | controller | CINDER_KUTTL_NAMESPACE: cinder-kuttl-tests 2026-03-09 18:34:05.699532 | controller | CINDER_REPO: https://github.com/openstack-k8s-operators/cinder-operator.git 2026-03-09 18:34:05.699537 | controller | CLEANUP_DIR_CMD: rm -Rf 2026-03-09 18:34:05.699541 | controller | CRC_BGP_NIC_1_MAC: '52:54:00:11:11:11' 2026-03-09 18:34:05.699545 | controller | CRC_BGP_NIC_2_MAC: '52:54:00:11:11:12' 2026-03-09 18:34:05.699550 | controller | CRC_HTTPS_PROXY: '' 2026-03-09 18:34:05.699554 | controller | CRC_HTTP_PROXY: '' 2026-03-09 18:34:05.699559 | controller | CRC_STORAGE_NAMESPACE: crc-storage 2026-03-09 18:34:05.699563 | controller | CRC_STORAGE_RETRIES: '3' 2026-03-09 18:34:05.699567 | controller | CRC_URL: '''https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/latest/crc-linux-amd64.tar.xz''' 2026-03-09 18:34:05.699572 | controller | CRC_VERSION: latest 2026-03-09 18:34:05.699576 | controller | DATAPLANE_ANSIBLE_SECRET: dataplane-ansible-ssh-private-key-secret 2026-03-09 18:34:05.699580 | controller | DATAPLANE_ANSIBLE_USER: '' 2026-03-09 18:34:05.699585 | controller | DATAPLANE_COMPUTE_IP: 192.168.122.100 2026-03-09 18:34:05.699590 | controller | DATAPLANE_CONTAINER_PREFIX: openstack 2026-03-09 18:34:05.699599 | controller | DATAPLANE_CONTAINER_TAG: current-podified 2026-03-09 18:34:05.699606 | controller | DATAPLANE_CUSTOM_SERVICE_RUNNER_IMG: quay.io/openstack-k8s-operators/openstack-ansibleee-runner:latest 2026-03-09 18:34:05.699612 | controller | DATAPLANE_DEFAULT_GW: 192.168.122.1 2026-03-09 18:34:05.699618 | controller | DATAPLANE_EXTRA_NOVA_CONFIG_FILE: /dev/null 2026-03-09 18:34:05.699625 | controller | DATAPLANE_GROWVOLS_ARGS: /=8GB /tmp=1GB /home=1GB /var=100% 2026-03-09 18:34:05.699631 | controller | DATAPLANE_KUSTOMIZE_SCENARIO: preprovisioned 2026-03-09 18:34:05.699637 | controller | DATAPLANE_NETWORKER_IP: 192.168.122.200 2026-03-09 18:34:05.699643 | controller | DATAPLANE_NETWORK_INTERFACE_NAME: eth0 2026-03-09 18:34:05.699652 | controller | DATAPLANE_NOVA_NFS_PATH: '' 2026-03-09 18:34:05.699656 | controller | DATAPLANE_NTP_SERVER: pool.ntp.org 2026-03-09 18:34:05.699661 | controller | DATAPLANE_PLAYBOOK: osp.edpm.download_cache 2026-03-09 18:34:05.699665 | controller | DATAPLANE_REGISTRY_URL: quay.io/podified-antelope-centos9 2026-03-09 18:34:05.699669 | controller | DATAPLANE_RUNNER_IMG: '' 2026-03-09 18:34:05.699674 | controller | DATAPLANE_SERVER_ROLE: compute 2026-03-09 18:34:05.699678 | controller | DATAPLANE_SSHD_ALLOWED_RANGES: '[''192.168.122.0/24'']' 2026-03-09 18:34:05.699682 | controller | DATAPLANE_TIMEOUT: 30m 2026-03-09 18:34:05.699687 | controller | DATAPLANE_TLS_ENABLED: 'true' 2026-03-09 18:34:05.699691 | controller | DATAPLANE_TOTAL_NETWORKER_NODES: '1' 2026-03-09 18:34:05.699695 | controller | DATAPLANE_TOTAL_NODES: '1' 2026-03-09 18:34:05.699700 | controller | DBSERVICE: galera 2026-03-09 18:34:05.699704 | controller | DESIGNATE: config/samples/designate_v1beta1_designate.yaml 2026-03-09 18:34:05.699708 | controller | DESIGNATE_BRANCH: main 2026-03-09 18:34:05.699713 | controller | DESIGNATE_COMMIT_HASH: '' 2026-03-09 18:34:05.699717 | controller | DESIGNATE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/config/samples/designate_v1beta1_designate.yaml 2026-03-09 18:34:05.699721 | controller | DESIGNATE_IMG: quay.io/openstack-k8s-operators/designate-operator-index:latest 2026-03-09 18:34:05.699726 | controller | DESIGNATE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/kuttl-test.yaml 2026-03-09 18:34:05.699730 | controller | DESIGNATE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/test/kuttl/tests 2026-03-09 18:34:05.699735 | controller | DESIGNATE_KUTTL_NAMESPACE: designate-kuttl-tests 2026-03-09 18:34:05.699739 | controller | DESIGNATE_REPO: https://github.com/openstack-k8s-operators/designate-operator.git 2026-03-09 18:34:05.699743 | controller | DNSDATA: config/samples/network_v1beta1_dnsdata.yaml 2026-03-09 18:34:05.699748 | controller | DNSDATA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsdata.yaml 2026-03-09 18:34:05.699752 | controller | DNSMASQ: config/samples/network_v1beta1_dnsmasq.yaml 2026-03-09 18:34:05.699757 | controller | DNSMASQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsmasq.yaml 2026-03-09 18:34:05.699761 | controller | DNS_DEPL_IMG: unused 2026-03-09 18:34:05.699765 | controller | DNS_DOMAIN: localdomain 2026-03-09 18:34:05.699773 | controller | DOWNLOAD_TOOLS_SELECTION: all 2026-03-09 18:34:05.699778 | controller | EDPM_ATTACH_EXTNET: 'true' 2026-03-09 18:34:05.699782 | controller | EDPM_COMPUTE_ADDITIONAL_HOST_ROUTES: '''[]''' 2026-03-09 18:34:05.699786 | controller | EDPM_COMPUTE_ADDITIONAL_NETWORKS: '''[]''' 2026-03-09 18:34:05.699791 | controller | EDPM_COMPUTE_CELLS: '1' 2026-03-09 18:34:05.699795 | controller | EDPM_COMPUTE_CEPH_ENABLED: 'true' 2026-03-09 18:34:05.699800 | controller | EDPM_COMPUTE_CEPH_NOVA: 'true' 2026-03-09 18:34:05.699804 | controller | EDPM_COMPUTE_DHCP_AGENT_ENABLED: 'true' 2026-03-09 18:34:05.699808 | controller | EDPM_COMPUTE_SRIOV_ENABLED: 'true' 2026-03-09 18:34:05.699813 | controller | EDPM_COMPUTE_SUFFIX: '0' 2026-03-09 18:34:05.699817 | controller | EDPM_CONFIGURE_DEFAULT_ROUTE: 'true' 2026-03-09 18:34:05.699821 | controller | EDPM_CONFIGURE_HUGEPAGES: 'false' 2026-03-09 18:34:05.699826 | controller | EDPM_CONFIGURE_NETWORKING: 'true' 2026-03-09 18:34:05.699830 | controller | EDPM_FIRSTBOOT_EXTRA: /tmp/edpm-firstboot-extra 2026-03-09 18:34:05.699834 | controller | EDPM_NETWORKER_SUFFIX: '0' 2026-03-09 18:34:05.699839 | controller | EDPM_TOTAL_NETWORKERS: '1' 2026-03-09 18:34:05.699843 | controller | EDPM_TOTAL_NODES: '1' 2026-03-09 18:34:05.699847 | controller | GALERA_REPLICAS: '' 2026-03-09 18:34:05.699852 | controller | GENERATE_SSH_KEYS: 'true' 2026-03-09 18:34:05.699856 | controller | GIT_CLONE_OPTS: '' 2026-03-09 18:34:05.699860 | controller | GLANCE: config/samples/glance_v1beta1_glance.yaml 2026-03-09 18:34:05.699865 | controller | GLANCEAPI_DEPL_IMG: unused 2026-03-09 18:34:05.699869 | controller | GLANCE_BRANCH: main 2026-03-09 18:34:05.699874 | controller | GLANCE_COMMIT_HASH: '' 2026-03-09 18:34:05.699878 | controller | GLANCE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/config/samples/glance_v1beta1_glance.yaml 2026-03-09 18:34:05.699920 | controller | GLANCE_IMG: quay.io/openstack-k8s-operators/glance-operator-index:latest 2026-03-09 18:34:05.699925 | controller | GLANCE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/kuttl-test.yaml 2026-03-09 18:34:05.699930 | controller | GLANCE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/test/kuttl/tests 2026-03-09 18:34:05.699934 | controller | GLANCE_KUTTL_NAMESPACE: glance-kuttl-tests 2026-03-09 18:34:05.699939 | controller | GLANCE_REPO: https://github.com/openstack-k8s-operators/glance-operator.git 2026-03-09 18:34:05.699943 | controller | HEAT: config/samples/heat_v1beta1_heat.yaml 2026-03-09 18:34:05.699947 | controller | HEATAPI_DEPL_IMG: unused 2026-03-09 18:34:05.699952 | controller | HEATCFNAPI_DEPL_IMG: unused 2026-03-09 18:34:05.699956 | controller | HEATENGINE_DEPL_IMG: unused 2026-03-09 18:34:05.699961 | controller | HEAT_AUTH_ENCRYPTION_KEY: 767c3ed056cbaa3b9dfedb8c6f825bf0 2026-03-09 18:34:05.699967 | controller | HEAT_BRANCH: main 2026-03-09 18:34:05.699972 | controller | HEAT_COMMIT_HASH: '' 2026-03-09 18:34:05.699977 | controller | HEAT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/config/samples/heat_v1beta1_heat.yaml 2026-03-09 18:34:05.699981 | controller | HEAT_IMG: quay.io/openstack-k8s-operators/heat-operator-index:latest 2026-03-09 18:34:05.699986 | controller | HEAT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/kuttl-test.yaml 2026-03-09 18:34:05.699991 | controller | HEAT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/test/kuttl/tests 2026-03-09 18:34:05.699996 | controller | HEAT_KUTTL_NAMESPACE: heat-kuttl-tests 2026-03-09 18:34:05.700000 | controller | HEAT_REPO: https://github.com/openstack-k8s-operators/heat-operator.git 2026-03-09 18:34:05.700005 | controller | HEAT_SERVICE_ENABLED: 'true' 2026-03-09 18:34:05.700009 | controller | HORIZON: config/samples/horizon_v1beta1_horizon.yaml 2026-03-09 18:34:05.700015 | controller | HORIZON_BRANCH: main 2026-03-09 18:34:05.700020 | controller | HORIZON_COMMIT_HASH: '' 2026-03-09 18:34:05.700024 | controller | HORIZON_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/config/samples/horizon_v1beta1_horizon.yaml 2026-03-09 18:34:05.700029 | controller | HORIZON_DEPL_IMG: unused 2026-03-09 18:34:05.700033 | controller | HORIZON_IMG: quay.io/openstack-k8s-operators/horizon-operator-index:latest 2026-03-09 18:34:05.700037 | controller | HORIZON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/kuttl-test.yaml 2026-03-09 18:34:05.700042 | controller | HORIZON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/test/kuttl/tests 2026-03-09 18:34:05.700046 | controller | HORIZON_KUTTL_NAMESPACE: horizon-kuttl-tests 2026-03-09 18:34:05.700050 | controller | HORIZON_REPO: https://github.com/openstack-k8s-operators/horizon-operator.git 2026-03-09 18:34:05.700056 | controller | INFRA_BRANCH: main 2026-03-09 18:34:05.700060 | controller | INFRA_COMMIT_HASH: '' 2026-03-09 18:34:05.700064 | controller | INFRA_IMG: quay.io/openstack-k8s-operators/infra-operator-index:latest 2026-03-09 18:34:05.700069 | controller | INFRA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/kuttl-test.yaml 2026-03-09 18:34:05.700073 | controller | INFRA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/test/kuttl/tests 2026-03-09 18:34:05.700078 | controller | INFRA_KUTTL_NAMESPACE: infra-kuttl-tests 2026-03-09 18:34:05.700082 | controller | INFRA_REPO: https://github.com/openstack-k8s-operators/infra-operator.git 2026-03-09 18:34:05.700086 | controller | INSTALL_CERT_MANAGER: 'true' 2026-03-09 18:34:05.700091 | controller | INSTALL_NMSTATE: true || false 2026-03-09 18:34:05.700095 | controller | INSTALL_NNCP: true || false 2026-03-09 18:34:05.700099 | controller | INTERNALAPI_HOST_ROUTES: '' 2026-03-09 18:34:05.700104 | controller | IPV6_LAB_IPV4_NETWORK_IPADDRESS: 172.30.0.1/24 2026-03-09 18:34:05.700108 | controller | IPV6_LAB_IPV6_NETWORK_IPADDRESS: fd00:abcd:abcd:fc00::1/64 2026-03-09 18:34:05.700112 | controller | IPV6_LAB_LIBVIRT_STORAGE_POOL: default 2026-03-09 18:34:05.700119 | controller | IPV6_LAB_MANAGE_FIREWALLD: 'true' 2026-03-09 18:34:05.700123 | controller | IPV6_LAB_NAT64_HOST_IPV4: 172.30.0.2/24 2026-03-09 18:34:05.700128 | controller | IPV6_LAB_NAT64_HOST_IPV6: fd00:abcd:abcd:fc00::2/64 2026-03-09 18:34:05.700134 | controller | IPV6_LAB_NAT64_INSTANCE_NAME: nat64-router 2026-03-09 18:34:05.700141 | controller | IPV6_LAB_NAT64_IPV6_NETWORK: fd00:abcd:abcd:fc00::/64 2026-03-09 18:34:05.700188 | controller | IPV6_LAB_NAT64_TAYGA_DYNAMIC_POOL: 192.168.255.0/24 2026-03-09 18:34:05.700199 | controller | IPV6_LAB_NAT64_TAYGA_IPV4: 192.168.255.1 2026-03-09 18:34:05.700206 | controller | IPV6_LAB_NAT64_TAYGA_IPV6: fd00:abcd:abcd:fc00::3 2026-03-09 18:34:05.700210 | controller | IPV6_LAB_NAT64_TAYGA_IPV6_PREFIX: fd00:abcd:abcd:fcff::/96 2026-03-09 18:34:05.700215 | controller | IPV6_LAB_NAT64_UPDATE_PACKAGES: 'false' 2026-03-09 18:34:05.700219 | controller | IPV6_LAB_NETWORK_NAME: nat64 2026-03-09 18:34:05.700224 | controller | IPV6_LAB_SNO_CLUSTER_NETWORK: fd00:abcd:0::/48 2026-03-09 18:34:05.700228 | controller | IPV6_LAB_SNO_HOST_IP: fd00:abcd:abcd:fc00::11 2026-03-09 18:34:05.700232 | controller | IPV6_LAB_SNO_HOST_PREFIX: '64' 2026-03-09 18:34:05.700237 | controller | IPV6_LAB_SNO_INSTANCE_NAME: sno 2026-03-09 18:34:05.700241 | controller | IPV6_LAB_SNO_MACHINE_NETWORK: fd00:abcd:abcd:fc00::/64 2026-03-09 18:34:05.700245 | controller | IPV6_LAB_SNO_OCP_MIRROR_URL: https://mirror.openshift.com/pub/openshift-v4/clients/ocp 2026-03-09 18:34:05.700250 | controller | IPV6_LAB_SNO_OCP_VERSION: latest-4.14 2026-03-09 18:34:05.700254 | controller | IPV6_LAB_SNO_SERVICE_NETWORK: fd00:abcd:abcd:fc03::/112 2026-03-09 18:34:05.700258 | controller | IPV6_LAB_SSH_PUB_KEY: /home/zuul/.ssh/id_rsa.pub 2026-03-09 18:34:05.700263 | controller | IPV6_LAB_WORK_DIR: /home/zuul/.ipv6lab 2026-03-09 18:34:05.700267 | controller | IRONIC: config/samples/ironic_v1beta1_ironic.yaml 2026-03-09 18:34:05.700271 | controller | IRONICAPI_DEPL_IMG: unused 2026-03-09 18:34:05.700276 | controller | IRONICCON_DEPL_IMG: unused 2026-03-09 18:34:05.700280 | controller | IRONICINS_DEPL_IMG: unused 2026-03-09 18:34:05.700284 | controller | IRONICNAG_DEPL_IMG: unused 2026-03-09 18:34:05.700289 | controller | IRONICPXE_DEPL_IMG: unused 2026-03-09 18:34:05.700293 | controller | IRONIC_BRANCH: main 2026-03-09 18:34:05.700297 | controller | IRONIC_COMMIT_HASH: '' 2026-03-09 18:34:05.700302 | controller | IRONIC_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/config/samples/ironic_v1beta1_ironic.yaml 2026-03-09 18:34:05.700306 | controller | IRONIC_IMAGE: quay.io/metal3-io/ironic 2026-03-09 18:34:05.700310 | controller | IRONIC_IMAGE_TAG: release-24.1 2026-03-09 18:34:05.700315 | controller | IRONIC_IMG: quay.io/openstack-k8s-operators/ironic-operator-index:latest 2026-03-09 18:34:05.700319 | controller | IRONIC_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/kuttl-test.yaml 2026-03-09 18:34:05.700324 | controller | IRONIC_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/test/kuttl/tests 2026-03-09 18:34:05.700328 | controller | IRONIC_KUTTL_NAMESPACE: ironic-kuttl-tests 2026-03-09 18:34:05.700333 | controller | IRONIC_REPO: https://github.com/openstack-k8s-operators/ironic-operator.git 2026-03-09 18:34:05.700341 | controller | KEYSTONEAPI: config/samples/keystone_v1beta1_keystoneapi.yaml 2026-03-09 18:34:05.700357 | controller | KEYSTONEAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/config/samples/keystone_v1beta1_keystoneapi.yaml 2026-03-09 18:34:05.700366 | controller | KEYSTONEAPI_DEPL_IMG: unused 2026-03-09 18:34:05.700377 | controller | KEYSTONE_BRANCH: main 2026-03-09 18:34:05.700383 | controller | KEYSTONE_COMMIT_HASH: '' 2026-03-09 18:34:05.700390 | controller | KEYSTONE_FEDERATION_CLIENT_SECRET: COX8bmlKAWn56XCGMrKQJj7dgHNAOl6f 2026-03-09 18:34:05.700396 | controller | KEYSTONE_FEDERATION_CRYPTO_PASSPHRASE: openstack 2026-03-09 18:34:05.700400 | controller | KEYSTONE_IMG: quay.io/openstack-k8s-operators/keystone-operator-index:latest 2026-03-09 18:34:05.700405 | controller | KEYSTONE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/kuttl-test.yaml 2026-03-09 18:34:05.700409 | controller | KEYSTONE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/test/kuttl/tests 2026-03-09 18:34:05.700417 | controller | KEYSTONE_KUTTL_NAMESPACE: keystone-kuttl-tests 2026-03-09 18:34:05.700421 | controller | KEYSTONE_REPO: https://github.com/openstack-k8s-operators/keystone-operator.git 2026-03-09 18:34:05.700426 | controller | KUBEADMIN_PWD: '12345678' 2026-03-09 18:34:05.700431 | controller | LIBVIRT_SECRET: libvirt-secret 2026-03-09 18:34:05.700435 | controller | LOKI_DEPLOY_MODE: openshift-network 2026-03-09 18:34:05.700439 | controller | LOKI_DEPLOY_NAMESPACE: netobserv 2026-03-09 18:34:05.700453 | controller | LOKI_DEPLOY_SIZE: 1x.demo 2026-03-09 18:34:05.700457 | controller | LOKI_NAMESPACE: openshift-operators-redhat 2026-03-09 18:34:05.700462 | controller | LOKI_OPERATOR_GROUP: openshift-operators-redhat-loki 2026-03-09 18:34:05.700469 | controller | LOKI_SUBSCRIPTION: loki-operator 2026-03-09 18:34:05.700474 | controller | LVMS_CR: '1' 2026-03-09 18:34:05.700478 | controller | MANILA: config/samples/manila_v1beta1_manila.yaml 2026-03-09 18:34:05.700482 | controller | MANILAAPI_DEPL_IMG: unused 2026-03-09 18:34:05.700487 | controller | MANILASCH_DEPL_IMG: unused 2026-03-09 18:34:05.700491 | controller | MANILASHARE_DEPL_IMG: unused 2026-03-09 18:34:05.700495 | controller | MANILA_BRANCH: main 2026-03-09 18:34:05.700500 | controller | MANILA_COMMIT_HASH: '' 2026-03-09 18:34:05.700504 | controller | MANILA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/config/samples/manila_v1beta1_manila.yaml 2026-03-09 18:34:05.700509 | controller | MANILA_IMG: quay.io/openstack-k8s-operators/manila-operator-index:latest 2026-03-09 18:34:05.700513 | controller | MANILA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/kuttl-test.yaml 2026-03-09 18:34:05.700518 | controller | MANILA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/test/kuttl/tests 2026-03-09 18:34:05.700522 | controller | MANILA_KUTTL_NAMESPACE: manila-kuttl-tests 2026-03-09 18:34:05.700526 | controller | MANILA_REPO: https://github.com/openstack-k8s-operators/manila-operator.git 2026-03-09 18:34:05.700531 | controller | MANILA_SERVICE_ENABLED: 'true' 2026-03-09 18:34:05.700535 | controller | MARIADB: config/samples/mariadb_v1beta1_galera.yaml 2026-03-09 18:34:05.700539 | controller | MARIADB_BRANCH: main 2026-03-09 18:34:05.700544 | controller | MARIADB_CHAINSAW_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/config.yaml 2026-03-09 18:34:05.700553 | controller | MARIADB_CHAINSAW_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/tests 2026-03-09 18:34:05.700558 | controller | MARIADB_CHAINSAW_NAMESPACE: mariadb-chainsaw-tests 2026-03-09 18:34:05.700562 | controller | MARIADB_COMMIT_HASH: '' 2026-03-09 18:34:05.700567 | controller | MARIADB_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/config/samples/mariadb_v1beta1_galera.yaml 2026-03-09 18:34:05.700571 | controller | MARIADB_DEPL_IMG: unused 2026-03-09 18:34:05.700575 | controller | MARIADB_IMG: quay.io/openstack-k8s-operators/mariadb-operator-index:latest 2026-03-09 18:34:05.700580 | controller | MARIADB_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/kuttl-test.yaml 2026-03-09 18:34:05.700584 | controller | MARIADB_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/kuttl/tests 2026-03-09 18:34:05.700588 | controller | MARIADB_KUTTL_NAMESPACE: mariadb-kuttl-tests 2026-03-09 18:34:05.700593 | controller | MARIADB_REPO: https://github.com/openstack-k8s-operators/mariadb-operator.git 2026-03-09 18:34:05.700597 | controller | MEMCACHED: config/samples/memcached_v1beta1_memcached.yaml 2026-03-09 18:34:05.700602 | controller | MEMCACHED_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/memcached_v1beta1_memcached.yaml 2026-03-09 18:34:05.700606 | controller | MEMCACHED_DEPL_IMG: unused 2026-03-09 18:34:05.700612 | controller | METADATA_SHARED_SECRET: '1234567842' 2026-03-09 18:34:05.700616 | controller | METALLB_IPV6_POOL: fd00:aaaa::80-fd00:aaaa::90 2026-03-09 18:34:05.700621 | controller | METALLB_POOL: 192.168.122.80-192.168.122.90 2026-03-09 18:34:05.700631 | controller | MICROSHIFT: '0' 2026-03-09 18:34:05.700635 | controller | MIRROR_INSECURE: 'true' 2026-03-09 18:34:05.700640 | controller | MIRROR_NAMESPACE: openstack-mirror 2026-03-09 18:34:05.700644 | controller | MIRROR_REGISTRY_HOST: default-route-openshift-image-registry.apps-crc.testing 2026-03-09 18:34:05.700649 | controller | NAMESPACE: openstack 2026-03-09 18:34:05.700653 | controller | NETCONFIG: config/samples/network_v1beta1_netconfig.yaml 2026-03-09 18:34:05.700658 | controller | NETCONFIG_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_netconfig.yaml 2026-03-09 18:34:05.700662 | controller | NETCONFIG_DEPL_IMG: unused 2026-03-09 18:34:05.700666 | controller | NETOBSERV_DEPLOY_NAMESPACE: netobserv 2026-03-09 18:34:05.700671 | controller | NETOBSERV_NAMESPACE: openshift-netobserv-operator 2026-03-09 18:34:05.700675 | controller | NETOBSERV_OPERATOR_GROUP: openshift-netobserv-operator-net 2026-03-09 18:34:05.700679 | controller | NETOBSERV_SUBSCRIPTION: netobserv-operator 2026-03-09 18:34:05.700684 | controller | NETWORK_BGP: 'false' 2026-03-09 18:34:05.700688 | controller | NETWORK_DESIGNATE_ADDRESS_PREFIX: 172.28.0 2026-03-09 18:34:05.700692 | controller | NETWORK_DESIGNATE_EXT_ADDRESS_PREFIX: 172.50.0 2026-03-09 18:34:05.700696 | controller | NETWORK_INTERNALAPI_ADDRESS_PREFIX: 172.17.0 2026-03-09 18:34:05.700701 | controller | NETWORK_ISOLATION: 'true' 2026-03-09 18:34:05.700705 | controller | NETWORK_ISOLATION_INSTANCE_NAME: crc 2026-03-09 18:34:05.700709 | controller | NETWORK_ISOLATION_IPV4: 'true' 2026-03-09 18:34:05.700714 | controller | NETWORK_ISOLATION_IPV4_ADDRESS: 172.16.1.1/24 2026-03-09 18:34:05.700718 | controller | NETWORK_ISOLATION_IPV4_NAT: 'true' 2026-03-09 18:34:05.700722 | controller | NETWORK_ISOLATION_IPV6: 'false' 2026-03-09 18:34:05.700727 | controller | NETWORK_ISOLATION_IPV6_ADDRESS: fd00:aaaa::1/64 2026-03-09 18:34:05.700731 | controller | NETWORK_ISOLATION_IP_ADDRESS: 192.168.122.10 2026-03-09 18:34:05.700735 | controller | NETWORK_ISOLATION_MAC: '52:54:00:11:11:10' 2026-03-09 18:34:05.700740 | controller | NETWORK_ISOLATION_NETWORK_NAME: net-iso 2026-03-09 18:34:05.700744 | controller | NETWORK_ISOLATION_NET_NAME: default 2026-03-09 18:34:05.700748 | controller | NETWORK_ISOLATION_USE_DEFAULT_NETWORK: 'true' 2026-03-09 18:34:05.700753 | controller | NETWORK_MTU: '1500' 2026-03-09 18:34:05.700757 | controller | NETWORK_STORAGEMGMT_ADDRESS_PREFIX: 172.20.0 2026-03-09 18:34:05.700761 | controller | NETWORK_STORAGE_ADDRESS_PREFIX: 172.18.0 2026-03-09 18:34:05.700765 | controller | NETWORK_STORAGE_MACVLAN: '' 2026-03-09 18:34:05.700770 | controller | NETWORK_TENANT_ADDRESS_PREFIX: 172.19.0 2026-03-09 18:34:05.700774 | controller | NETWORK_VLAN_START: '20' 2026-03-09 18:34:05.700778 | controller | NETWORK_VLAN_STEP: '1' 2026-03-09 18:34:05.700783 | controller | NEUTRONAPI: config/samples/neutron_v1beta1_neutronapi.yaml 2026-03-09 18:34:05.700787 | controller | NEUTRONAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/config/samples/neutron_v1beta1_neutronapi.yaml 2026-03-09 18:34:05.700791 | controller | NEUTRONAPI_DEPL_IMG: unused 2026-03-09 18:34:05.700796 | controller | NEUTRON_BRANCH: main 2026-03-09 18:34:05.700800 | controller | NEUTRON_COMMIT_HASH: '' 2026-03-09 18:34:05.700804 | controller | NEUTRON_IMG: quay.io/openstack-k8s-operators/neutron-operator-index:latest 2026-03-09 18:34:05.700809 | controller | NEUTRON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/kuttl-test.yaml 2026-03-09 18:34:05.700814 | controller | NEUTRON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/test/kuttl/tests 2026-03-09 18:34:05.700818 | controller | NEUTRON_KUTTL_NAMESPACE: neutron-kuttl-tests 2026-03-09 18:34:05.700822 | controller | NEUTRON_REPO: https://github.com/openstack-k8s-operators/neutron-operator.git 2026-03-09 18:34:05.700827 | controller | NFS_HOME: /home/nfs 2026-03-09 18:34:05.700831 | controller | NMSTATE_NAMESPACE: openshift-nmstate 2026-03-09 18:34:05.700835 | controller | NMSTATE_OPERATOR_GROUP: openshift-nmstate-tn6k8 2026-03-09 18:34:05.700840 | controller | NMSTATE_SUBSCRIPTION: kubernetes-nmstate-operator 2026-03-09 18:34:05.700844 | controller | NNCP_ADDITIONAL_HOST_ROUTES: '' 2026-03-09 18:34:05.700848 | controller | NNCP_BGP_1_INTERFACE: enp7s0 2026-03-09 18:34:05.700855 | controller | NNCP_BGP_1_IP_ADDRESS: 100.65.4.2 2026-03-09 18:34:05.700859 | controller | NNCP_BGP_2_INTERFACE: enp8s0 2026-03-09 18:34:05.700863 | controller | NNCP_BGP_2_IP_ADDRESS: 100.64.4.2 2026-03-09 18:34:05.700868 | controller | NNCP_BRIDGE: ospbr 2026-03-09 18:34:05.700872 | controller | NNCP_CLEANUP_TIMEOUT: 120s 2026-03-09 18:34:05.700876 | controller | NNCP_CTLPLANE_IPV6_ADDRESS_PREFIX: 'fd00:aaaa::' 2026-03-09 18:34:05.700881 | controller | NNCP_CTLPLANE_IPV6_ADDRESS_SUFFIX: '10' 2026-03-09 18:34:05.700888 | controller | NNCP_CTLPLANE_IP_ADDRESS_PREFIX: 192.168.122 2026-03-09 18:34:05.700892 | controller | NNCP_CTLPLANE_IP_ADDRESS_SUFFIX: '10' 2026-03-09 18:34:05.700896 | controller | NNCP_DNS_SERVER: 192.168.122.1 2026-03-09 18:34:05.700901 | controller | NNCP_DNS_SERVER_IPV6: fd00:aaaa::1 2026-03-09 18:34:05.700905 | controller | NNCP_GATEWAY: 192.168.122.1 2026-03-09 18:34:05.700909 | controller | NNCP_GATEWAY_IPV6: fd00:aaaa::1 2026-03-09 18:34:05.700914 | controller | NNCP_INTERFACE: enp6s0 2026-03-09 18:34:05.700918 | controller | NNCP_NODES: '' 2026-03-09 18:34:05.700922 | controller | NNCP_TIMEOUT: 240s 2026-03-09 18:34:05.700927 | controller | NOVA: config/samples/nova_v1beta1_nova_collapsed_cell.yaml 2026-03-09 18:34:05.700931 | controller | NOVA_BRANCH: main 2026-03-09 18:34:05.700935 | controller | NOVA_COMMIT_HASH: '' 2026-03-09 18:34:05.700940 | controller | NOVA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/nova-operator/config/samples/nova_v1beta1_nova_collapsed_cell.yaml 2026-03-09 18:34:05.700944 | controller | NOVA_IMG: quay.io/openstack-k8s-operators/nova-operator-index:latest 2026-03-09 18:34:05.700949 | controller | NOVA_REPO: https://github.com/openstack-k8s-operators/nova-operator.git 2026-03-09 18:34:05.700953 | controller | NUMBER_OF_INSTANCES: '1' 2026-03-09 18:34:05.700958 | controller | OCP_NETWORK_NAME: crc 2026-03-09 18:34:05.700962 | controller | OCTAVIA: config/samples/octavia_v1beta1_octavia.yaml 2026-03-09 18:34:05.700966 | controller | OCTAVIA_BRANCH: main 2026-03-09 18:34:05.700972 | controller | OCTAVIA_COMMIT_HASH: '' 2026-03-09 18:34:05.700977 | controller | OCTAVIA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/config/samples/octavia_v1beta1_octavia.yaml 2026-03-09 18:34:05.700982 | controller | OCTAVIA_IMG: quay.io/openstack-k8s-operators/octavia-operator-index:latest 2026-03-09 18:34:05.700987 | controller | OCTAVIA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/kuttl-test.yaml 2026-03-09 18:34:05.700993 | controller | OCTAVIA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/test/kuttl/tests 2026-03-09 18:34:05.700998 | controller | OCTAVIA_KUTTL_NAMESPACE: octavia-kuttl-tests 2026-03-09 18:34:05.701003 | controller | OCTAVIA_REPO: https://github.com/openstack-k8s-operators/octavia-operator.git 2026-03-09 18:34:05.701007 | controller | OKD: 'false' 2026-03-09 18:34:05.701012 | controller | OPENSTACK_BRANCH: main 2026-03-09 18:34:05.701016 | controller | OPENSTACK_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-bundle:latest 2026-03-09 18:34:05.701020 | controller | OPENSTACK_COMMIT_HASH: '' 2026-03-09 18:34:05.701025 | controller | OPENSTACK_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml 2026-03-09 18:34:05.701029 | controller | OPENSTACK_CRDS_DIR: openstack_crds 2026-03-09 18:34:05.701034 | controller | OPENSTACK_CTLPLANE: config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml 2026-03-09 18:34:05.701038 | controller | OPENSTACK_IMG: quay.io/openstack-k8s-operators/openstack-operator-index:latest 2026-03-09 18:34:05.701043 | controller | OPENSTACK_K8S_BRANCH: main 2026-03-09 18:34:05.701047 | controller | OPENSTACK_K8S_TAG: latest 2026-03-09 18:34:05.701051 | controller | OPENSTACK_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/kuttl-test.yaml 2026-03-09 18:34:05.701056 | controller | OPENSTACK_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/test/kuttl/tests 2026-03-09 18:34:05.701060 | controller | OPENSTACK_KUTTL_NAMESPACE: openstack-kuttl-tests 2026-03-09 18:34:05.701064 | controller | OPENSTACK_NEUTRON_CUSTOM_CONF: '' 2026-03-09 18:34:05.701071 | controller | OPENSTACK_REPO: https://github.com/openstack-k8s-operators/openstack-operator.git 2026-03-09 18:34:05.701076 | controller | OPENSTACK_STORAGE_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-storage-bundle:latest 2026-03-09 18:34:05.701080 | controller | OPERATOR_BASE_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator 2026-03-09 18:34:05.701085 | controller | OPERATOR_CHANNEL: '' 2026-03-09 18:34:05.701089 | controller | OPERATOR_NAMESPACE: openstack-operators 2026-03-09 18:34:05.701093 | controller | OPERATOR_SOURCE: '' 2026-03-09 18:34:05.701098 | controller | OPERATOR_SOURCE_NAMESPACE: '' 2026-03-09 18:34:05.701102 | controller | OUT: /home/zuul/ci-framework-data/artifacts/manifests 2026-03-09 18:34:05.701107 | controller | OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm 2026-03-09 18:34:05.701111 | controller | OVNCONTROLLER: config/samples/ovn_v1beta1_ovncontroller.yaml 2026-03-09 18:34:05.701115 | controller | OVNCONTROLLER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovncontroller.yaml 2026-03-09 18:34:05.701120 | controller | OVNCONTROLLER_NMAP: 'true' 2026-03-09 18:34:05.701124 | controller | OVNDBS: config/samples/ovn_v1beta1_ovndbcluster.yaml 2026-03-09 18:34:05.701130 | controller | OVNDBS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovndbcluster.yaml 2026-03-09 18:34:05.701135 | controller | OVNNORTHD: config/samples/ovn_v1beta1_ovnnorthd.yaml 2026-03-09 18:34:05.701139 | controller | OVNNORTHD_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovnnorthd.yaml 2026-03-09 18:34:05.701143 | controller | OVN_BRANCH: main 2026-03-09 18:34:05.701165 | controller | OVN_COMMIT_HASH: '' 2026-03-09 18:34:05.701173 | controller | OVN_IMG: quay.io/openstack-k8s-operators/ovn-operator-index:latest 2026-03-09 18:34:05.701178 | controller | OVN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/kuttl-test.yaml 2026-03-09 18:34:05.701184 | controller | OVN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/test/kuttl/tests 2026-03-09 18:34:05.701189 | controller | OVN_KUTTL_NAMESPACE: ovn-kuttl-tests 2026-03-09 18:34:05.701193 | controller | OVN_REPO: https://github.com/openstack-k8s-operators/ovn-operator.git 2026-03-09 18:34:05.701197 | controller | PASSWORD: '12345678' 2026-03-09 18:34:05.701205 | controller | PLACEMENTAPI: config/samples/placement_v1beta1_placementapi.yaml 2026-03-09 18:34:05.706732 | controller | PLACEMENTAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/config/samples/placement_v1beta1_placementapi.yaml 2026-03-09 18:34:05.706750 | controller | PLACEMENTAPI_DEPL_IMG: unused 2026-03-09 18:34:05.706755 | controller | PLACEMENT_BRANCH: main 2026-03-09 18:34:05.706760 | controller | PLACEMENT_COMMIT_HASH: '' 2026-03-09 18:34:05.706769 | controller | PLACEMENT_IMG: quay.io/openstack-k8s-operators/placement-operator-index:latest 2026-03-09 18:34:05.706773 | controller | PLACEMENT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/kuttl-test.yaml 2026-03-09 18:34:05.706778 | controller | PLACEMENT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/test/kuttl/tests 2026-03-09 18:34:05.706782 | controller | PLACEMENT_KUTTL_NAMESPACE: placement-kuttl-tests 2026-03-09 18:34:05.706786 | controller | PLACEMENT_REPO: https://github.com/openstack-k8s-operators/placement-operator.git 2026-03-09 18:34:05.706791 | controller | PULL_SECRET: /home/zuul/pull-secret.txt 2026-03-09 18:34:05.706796 | controller | RABBITMQ: docs/examples/default-security-context/rabbitmq.yaml 2026-03-09 18:34:05.706800 | controller | RABBITMQ_BRANCH: patches 2026-03-09 18:34:05.706805 | controller | RABBITMQ_COMMIT_HASH: '' 2026-03-09 18:34:05.706809 | controller | RABBITMQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rabbitmq-operator/docs/examples/default-security-context/rabbitmq.yaml 2026-03-09 18:34:05.706814 | controller | RABBITMQ_DEPL_IMG: unused 2026-03-09 18:34:05.706818 | controller | RABBITMQ_IMG: quay.io/openstack-k8s-operators/rabbitmq-cluster-operator-index:latest 2026-03-09 18:34:05.706825 | controller | RABBITMQ_REPO: https://github.com/openstack-k8s-operators/rabbitmq-cluster-operator.git 2026-03-09 18:34:05.706830 | controller | REDHAT_OPERATORS: 'false' 2026-03-09 18:34:05.706834 | controller | REDIS: config/samples/redis_v1beta1_redis.yaml 2026-03-09 18:34:05.706839 | controller | REDIS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator-redis/config/samples/redis_v1beta1_redis.yaml 2026-03-09 18:34:05.706843 | controller | REDIS_DEPL_IMG: unused 2026-03-09 18:34:05.706847 | controller | RH_REGISTRY_PWD: '' 2026-03-09 18:34:05.706852 | controller | RH_REGISTRY_USER: '' 2026-03-09 18:34:05.706856 | controller | SECRET: osp-secret 2026-03-09 18:34:05.706860 | controller | SG_CORE_DEPL_IMG: unused 2026-03-09 18:34:05.706865 | controller | STANDALONE_COMPUTE_DRIVER: libvirt 2026-03-09 18:34:05.706869 | controller | STANDALONE_EXTERNAL_NET_PREFFIX: 172.21.0 2026-03-09 18:34:05.706873 | controller | STANDALONE_INTERNALAPI_NET_PREFIX: 172.17.0 2026-03-09 18:34:05.706878 | controller | STANDALONE_STORAGEMGMT_NET_PREFIX: 172.20.0 2026-03-09 18:34:05.706882 | controller | STANDALONE_STORAGE_NET_PREFIX: 172.18.0 2026-03-09 18:34:05.706886 | controller | STANDALONE_TENANT_NET_PREFIX: 172.19.0 2026-03-09 18:34:05.706891 | controller | STORAGEMGMT_HOST_ROUTES: '' 2026-03-09 18:34:05.706895 | controller | STORAGE_CLASS: local-storage 2026-03-09 18:34:05.706899 | controller | STORAGE_HOST_ROUTES: '' 2026-03-09 18:34:05.706904 | controller | SWIFT: config/samples/swift_v1beta1_swift.yaml 2026-03-09 18:34:05.706908 | controller | SWIFT_BRANCH: main 2026-03-09 18:34:05.706912 | controller | SWIFT_COMMIT_HASH: '' 2026-03-09 18:34:05.706917 | controller | SWIFT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/config/samples/swift_v1beta1_swift.yaml 2026-03-09 18:34:05.706921 | controller | SWIFT_IMG: quay.io/openstack-k8s-operators/swift-operator-index:latest 2026-03-09 18:34:05.706925 | controller | SWIFT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/kuttl-test.yaml 2026-03-09 18:34:05.706930 | controller | SWIFT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/test/kuttl/tests 2026-03-09 18:34:05.706934 | controller | SWIFT_KUTTL_NAMESPACE: swift-kuttl-tests 2026-03-09 18:34:05.706939 | controller | SWIFT_REPO: https://github.com/openstack-k8s-operators/swift-operator.git 2026-03-09 18:34:05.706943 | controller | TELEMETRY: config/samples/telemetry_v1beta1_telemetry.yaml 2026-03-09 18:34:05.706948 | controller | TELEMETRY_BRANCH: main 2026-03-09 18:34:05.706954 | controller | TELEMETRY_COMMIT_HASH: '' 2026-03-09 18:34:05.706958 | controller | TELEMETRY_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/config/samples/telemetry_v1beta1_telemetry.yaml 2026-03-09 18:34:05.706963 | controller | TELEMETRY_IMG: quay.io/openstack-k8s-operators/telemetry-operator-index:latest 2026-03-09 18:34:05.706969 | controller | TELEMETRY_KUTTL_BASEDIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator 2026-03-09 18:34:05.706974 | controller | TELEMETRY_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/kuttl-test.yaml 2026-03-09 18:34:05.706978 | controller | TELEMETRY_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/test/kuttl/suites 2026-03-09 18:34:05.706983 | controller | TELEMETRY_KUTTL_NAMESPACE: telemetry-kuttl-tests 2026-03-09 18:34:05.706987 | controller | TELEMETRY_KUTTL_RELPATH: test/kuttl/suites 2026-03-09 18:34:05.706991 | controller | TELEMETRY_REPO: https://github.com/openstack-k8s-operators/telemetry-operator.git 2026-03-09 18:34:05.706996 | controller | TENANT_HOST_ROUTES: '' 2026-03-09 18:34:05.707000 | controller | TIMEOUT: 300s 2026-03-09 18:34:05.707005 | controller | TLS_ENABLED: 'false' 2026-03-09 18:34:05.707009 | controller | WATCHER_BRANCH: '' 2026-03-09 18:34:05.707013 | controller | WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator 2026-03-09 18:34:05.707018 | controller | tripleo_deploy: 'export REGISTRY_USER:' 2026-03-09 18:34:05.707025 | controller | 2026-03-09 18:34:06.045269 | controller | TASK [install_yamls : Generate make targets install_yamls_path={{ cifmw_install_yamls_repo }}, output_directory={{ cifmw_install_yamls_tasks_out }}] *** 2026-03-09 18:34:06.045332 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.035) 0:01:01.641 ********** 2026-03-09 18:34:06.045339 | controller | Monday 09 March 2026 18:34:05 +0000 (0:00:00.035) 0:01:01.640 ********** 2026-03-09 18:34:06.045352 | controller | ok: [controller] 2026-03-09 18:34:06.062482 | controller | 2026-03-09 18:34:06.062537 | controller | TASK [install_yamls : Debug generate_make module var=cifmw_generate_makes] ***** 2026-03-09 18:34:06.062545 | controller | Monday 09 March 2026 18:34:06 +0000 (0:00:00.338) 0:01:01.980 ********** 2026-03-09 18:34:06.062559 | controller | Monday 09 March 2026 18:34:06 +0000 (0:00:00.338) 0:01:01.978 ********** 2026-03-09 18:34:06.062572 | controller | ok: [controller] => 2026-03-09 18:34:06.062579 | controller | cifmw_generate_makes: 2026-03-09 18:34:06.062585 | controller | changed: false 2026-03-09 18:34:06.062593 | controller | debug: 2026-03-09 18:34:06.062599 | controller | /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/Makefile: 2026-03-09 18:34:06.062605 | controller | - all 2026-03-09 18:34:06.062611 | controller | - help 2026-03-09 18:34:06.062617 | controller | - cleanup 2026-03-09 18:34:06.062622 | controller | - deploy_cleanup 2026-03-09 18:34:06.062629 | controller | - wait 2026-03-09 18:34:06.062635 | controller | - crc_storage 2026-03-09 18:34:06.062641 | controller | - crc_storage_cleanup 2026-03-09 18:34:06.062647 | controller | - crc_storage_release 2026-03-09 18:34:06.062653 | controller | - crc_storage_with_retries 2026-03-09 18:34:06.062659 | controller | - crc_storage_cleanup_with_retries 2026-03-09 18:34:06.062665 | controller | - operator_namespace 2026-03-09 18:34:06.062671 | controller | - namespace 2026-03-09 18:34:06.062676 | controller | - namespace_cleanup 2026-03-09 18:34:06.062682 | controller | - input 2026-03-09 18:34:06.062688 | controller | - input_cleanup 2026-03-09 18:34:06.062693 | controller | - crc_bmo_setup 2026-03-09 18:34:06.062697 | controller | - crc_bmo_cleanup 2026-03-09 18:34:06.062702 | controller | - mirror_registry 2026-03-09 18:34:06.062706 | controller | - mirror_registry_secure 2026-03-09 18:34:06.062711 | controller | - mirror_registry_ca 2026-03-09 18:34:06.062715 | controller | - mirror_registry_cleanup 2026-03-09 18:34:06.062720 | controller | - openstack_prep 2026-03-09 18:34:06.062724 | controller | - openstack 2026-03-09 18:34:06.062728 | controller | - openstack_wait 2026-03-09 18:34:06.062733 | controller | - openstack_init 2026-03-09 18:34:06.062737 | controller | - openstack_cleanup 2026-03-09 18:34:06.062741 | controller | - openstack_repo 2026-03-09 18:34:06.062745 | controller | - openstack_deploy_prep 2026-03-09 18:34:06.062750 | controller | - openstack_deploy 2026-03-09 18:34:06.062754 | controller | - openstack_wait_deploy 2026-03-09 18:34:06.062758 | controller | - openstack_deploy_cleanup 2026-03-09 18:34:06.062763 | controller | - openstack_update_run 2026-03-09 18:34:06.062767 | controller | - update_services 2026-03-09 18:34:06.062772 | controller | - update_system 2026-03-09 18:34:06.062776 | controller | - openstack_patch_version 2026-03-09 18:34:06.062781 | controller | - edpm_deploy_generate_keys 2026-03-09 18:34:06.062785 | controller | - edpm_patch_ansible_runner_image 2026-03-09 18:34:06.062789 | controller | - edpm_deploy_prep 2026-03-09 18:34:06.062794 | controller | - edpm_deploy_cleanup 2026-03-09 18:34:06.062798 | controller | - edpm_deploy 2026-03-09 18:34:06.062802 | controller | - edpm_deploy_baremetal_prep 2026-03-09 18:34:06.062807 | controller | - edpm_deploy_baremetal 2026-03-09 18:34:06.062811 | controller | - edpm_wait_deploy_baremetal 2026-03-09 18:34:06.062815 | controller | - edpm_wait_deploy 2026-03-09 18:34:06.062819 | controller | - edpm_register_dns 2026-03-09 18:34:06.062824 | controller | - edpm_nova_discover_hosts 2026-03-09 18:34:06.062828 | controller | - openstack_crds 2026-03-09 18:34:06.062832 | controller | - openstack_crds_cleanup 2026-03-09 18:34:06.062837 | controller | - edpm_deploy_networker_prep 2026-03-09 18:34:06.062841 | controller | - edpm_deploy_networker_cleanup 2026-03-09 18:34:06.062845 | controller | - edpm_deploy_networker 2026-03-09 18:34:06.062850 | controller | - infra_prep 2026-03-09 18:34:06.062854 | controller | - infra 2026-03-09 18:34:06.062858 | controller | - infra_cleanup 2026-03-09 18:34:06.062862 | controller | - dns_deploy_prep 2026-03-09 18:34:06.062871 | controller | - dns_deploy 2026-03-09 18:34:06.062876 | controller | - dns_deploy_cleanup 2026-03-09 18:34:06.062880 | controller | - netconfig_deploy_prep 2026-03-09 18:34:06.062884 | controller | - netconfig_deploy 2026-03-09 18:34:06.062888 | controller | - netconfig_deploy_cleanup 2026-03-09 18:34:06.062892 | controller | - memcached_deploy_prep 2026-03-09 18:34:06.062897 | controller | - memcached_deploy 2026-03-09 18:34:06.062901 | controller | - memcached_deploy_cleanup 2026-03-09 18:34:06.062905 | controller | - keystone_prep 2026-03-09 18:34:06.062910 | controller | - keystone 2026-03-09 18:34:06.062914 | controller | - keystone_cleanup 2026-03-09 18:34:06.062918 | controller | - keystone_deploy_prep 2026-03-09 18:34:06.062922 | controller | - keystone_deploy 2026-03-09 18:34:06.062926 | controller | - keystone_deploy_cleanup 2026-03-09 18:34:06.062931 | controller | - barbican_prep 2026-03-09 18:34:06.062935 | controller | - barbican 2026-03-09 18:34:06.062939 | controller | - barbican_cleanup 2026-03-09 18:34:06.062943 | controller | - barbican_deploy_prep 2026-03-09 18:34:06.062948 | controller | - barbican_deploy 2026-03-09 18:34:06.062952 | controller | - barbican_deploy_validate 2026-03-09 18:34:06.062956 | controller | - barbican_deploy_cleanup 2026-03-09 18:34:06.062960 | controller | - mariadb 2026-03-09 18:34:06.062965 | controller | - mariadb_cleanup 2026-03-09 18:34:06.062969 | controller | - mariadb_deploy_prep 2026-03-09 18:34:06.062973 | controller | - mariadb_deploy 2026-03-09 18:34:06.062977 | controller | - mariadb_deploy_cleanup 2026-03-09 18:34:06.062982 | controller | - placement_prep 2026-03-09 18:34:06.062986 | controller | - placement 2026-03-09 18:34:06.062990 | controller | - placement_cleanup 2026-03-09 18:34:06.062994 | controller | - placement_deploy_prep 2026-03-09 18:34:06.062998 | controller | - placement_deploy 2026-03-09 18:34:06.063003 | controller | - placement_deploy_cleanup 2026-03-09 18:34:06.063007 | controller | - glance_prep 2026-03-09 18:34:06.063011 | controller | - glance 2026-03-09 18:34:06.063015 | controller | - glance_cleanup 2026-03-09 18:34:06.063020 | controller | - glance_deploy_prep 2026-03-09 18:34:06.063024 | controller | - glance_deploy 2026-03-09 18:34:06.063028 | controller | - glance_deploy_cleanup 2026-03-09 18:34:06.063033 | controller | - ovn_prep 2026-03-09 18:34:06.063039 | controller | - ovn 2026-03-09 18:34:06.063044 | controller | - ovn_cleanup 2026-03-09 18:34:06.063048 | controller | - ovn_deploy_prep 2026-03-09 18:34:06.063053 | controller | - ovn_deploy 2026-03-09 18:34:06.063057 | controller | - ovn_deploy_cleanup 2026-03-09 18:34:06.063061 | controller | - neutron_prep 2026-03-09 18:34:06.063065 | controller | - neutron 2026-03-09 18:34:06.063070 | controller | - neutron_cleanup 2026-03-09 18:34:06.063074 | controller | - neutron_deploy_prep 2026-03-09 18:34:06.063078 | controller | - neutron_deploy 2026-03-09 18:34:06.063082 | controller | - neutron_deploy_cleanup 2026-03-09 18:34:06.063087 | controller | - cinder_prep 2026-03-09 18:34:06.063091 | controller | - cinder 2026-03-09 18:34:06.063095 | controller | - cinder_cleanup 2026-03-09 18:34:06.063099 | controller | - cinder_deploy_prep 2026-03-09 18:34:06.063104 | controller | - cinder_deploy 2026-03-09 18:34:06.063109 | controller | - cinder_deploy_cleanup 2026-03-09 18:34:06.063113 | controller | - rabbitmq_prep 2026-03-09 18:34:06.063117 | controller | - rabbitmq 2026-03-09 18:34:06.063121 | controller | - rabbitmq_cleanup 2026-03-09 18:34:06.063126 | controller | - rabbitmq_deploy_prep 2026-03-09 18:34:06.063130 | controller | - rabbitmq_deploy 2026-03-09 18:34:06.063134 | controller | - rabbitmq_deploy_cleanup 2026-03-09 18:34:06.063138 | controller | - ironic_prep 2026-03-09 18:34:06.063143 | controller | - ironic 2026-03-09 18:34:06.063162 | controller | - ironic_cleanup 2026-03-09 18:34:06.063169 | controller | - ironic_deploy_prep 2026-03-09 18:34:06.063173 | controller | - ironic_deploy 2026-03-09 18:34:06.063178 | controller | - ironic_deploy_cleanup 2026-03-09 18:34:06.063182 | controller | - octavia_prep 2026-03-09 18:34:06.063186 | controller | - octavia 2026-03-09 18:34:06.063190 | controller | - octavia_cleanup 2026-03-09 18:34:06.063194 | controller | - octavia_deploy_prep 2026-03-09 18:34:06.063199 | controller | - octavia_deploy 2026-03-09 18:34:06.063203 | controller | - octavia_deploy_cleanup 2026-03-09 18:34:06.063209 | controller | - designate_prep 2026-03-09 18:34:06.063213 | controller | - designate 2026-03-09 18:34:06.063218 | controller | - designate_cleanup 2026-03-09 18:34:06.063222 | controller | - designate_deploy_prep 2026-03-09 18:34:06.063226 | controller | - designate_deploy 2026-03-09 18:34:06.063230 | controller | - designate_deploy_cleanup 2026-03-09 18:34:06.063234 | controller | - nova_prep 2026-03-09 18:34:06.063239 | controller | - nova 2026-03-09 18:34:06.063243 | controller | - nova_cleanup 2026-03-09 18:34:06.063247 | controller | - nova_deploy_prep 2026-03-09 18:34:06.063251 | controller | - nova_deploy 2026-03-09 18:34:06.063256 | controller | - nova_deploy_cleanup 2026-03-09 18:34:06.063260 | controller | - mariadb_kuttl_run 2026-03-09 18:34:06.063264 | controller | - mariadb_kuttl 2026-03-09 18:34:06.063268 | controller | - kuttl_db_prep 2026-03-09 18:34:06.063273 | controller | - kuttl_db_cleanup 2026-03-09 18:34:06.063277 | controller | - kuttl_common_prep 2026-03-09 18:34:06.063286 | controller | - kuttl_common_cleanup 2026-03-09 18:34:06.063290 | controller | - keystone_kuttl_run 2026-03-09 18:34:06.063295 | controller | - keystone_kuttl 2026-03-09 18:34:06.063299 | controller | - barbican_kuttl_run 2026-03-09 18:34:06.063303 | controller | - barbican_kuttl 2026-03-09 18:34:06.063307 | controller | - placement_kuttl_run 2026-03-09 18:34:06.063311 | controller | - placement_kuttl 2026-03-09 18:34:06.063316 | controller | - cinder_kuttl_run 2026-03-09 18:34:06.063320 | controller | - cinder_kuttl 2026-03-09 18:34:06.063324 | controller | - neutron_kuttl_run 2026-03-09 18:34:06.063328 | controller | - neutron_kuttl 2026-03-09 18:34:06.063332 | controller | - octavia_kuttl_run 2026-03-09 18:34:06.063337 | controller | - octavia_kuttl 2026-03-09 18:34:06.063341 | controller | - designate_kuttl 2026-03-09 18:34:06.063345 | controller | - designate_kuttl_run 2026-03-09 18:34:06.063349 | controller | - ovn_kuttl_run 2026-03-09 18:34:06.063353 | controller | - ovn_kuttl 2026-03-09 18:34:06.063358 | controller | - infra_kuttl_run 2026-03-09 18:34:06.063362 | controller | - infra_kuttl 2026-03-09 18:34:06.063366 | controller | - ironic_kuttl_run 2026-03-09 18:34:06.063372 | controller | - ironic_kuttl 2026-03-09 18:34:06.063376 | controller | - ironic_kuttl_crc 2026-03-09 18:34:06.063380 | controller | - heat_kuttl_run 2026-03-09 18:34:06.063385 | controller | - heat_kuttl 2026-03-09 18:34:06.063389 | controller | - heat_kuttl_crc 2026-03-09 18:34:06.063393 | controller | - glance_kuttl_run 2026-03-09 18:34:06.063397 | controller | - glance_kuttl 2026-03-09 18:34:06.063402 | controller | - manila_kuttl_run 2026-03-09 18:34:06.063406 | controller | - manila_kuttl 2026-03-09 18:34:06.063410 | controller | - swift_kuttl_run 2026-03-09 18:34:06.063414 | controller | - swift_kuttl 2026-03-09 18:34:06.063418 | controller | - horizon_kuttl_run 2026-03-09 18:34:06.063423 | controller | - horizon_kuttl 2026-03-09 18:34:06.063427 | controller | - openstack_kuttl_prep 2026-03-09 18:34:06.063431 | controller | - openstack_kuttl_run 2026-03-09 18:34:06.063435 | controller | - openstack_kuttl_cleanup 2026-03-09 18:34:06.063439 | controller | - openstack_kuttl 2026-03-09 18:34:06.063444 | controller | - mariadb_chainsaw_run 2026-03-09 18:34:06.063448 | controller | - mariadb_chainsaw 2026-03-09 18:34:06.063452 | controller | - horizon_prep 2026-03-09 18:34:06.063456 | controller | - horizon 2026-03-09 18:34:06.063461 | controller | - horizon_cleanup 2026-03-09 18:34:06.063465 | controller | - horizon_deploy_prep 2026-03-09 18:34:06.063469 | controller | - horizon_deploy 2026-03-09 18:34:06.063473 | controller | - horizon_deploy_cleanup 2026-03-09 18:34:06.063478 | controller | - heat_prep 2026-03-09 18:34:06.063482 | controller | - heat 2026-03-09 18:34:06.063486 | controller | - heat_cleanup 2026-03-09 18:34:06.063490 | controller | - heat_deploy_prep 2026-03-09 18:34:06.063495 | controller | - heat_deploy 2026-03-09 18:34:06.063499 | controller | - heat_deploy_cleanup 2026-03-09 18:34:06.063503 | controller | - baremetal_prep 2026-03-09 18:34:06.063507 | controller | - baremetal 2026-03-09 18:34:06.063511 | controller | - baremetal_cleanup 2026-03-09 18:34:06.063516 | controller | - ceph_help 2026-03-09 18:34:06.063520 | controller | - ceph 2026-03-09 18:34:06.063524 | controller | - ceph_cleanup 2026-03-09 18:34:06.063528 | controller | - rook_prep 2026-03-09 18:34:06.063532 | controller | - rook 2026-03-09 18:34:06.063557 | controller | - rook_deploy_prep 2026-03-09 18:34:06.063562 | controller | - rook_deploy 2026-03-09 18:34:06.063566 | controller | - rook_crc_disk 2026-03-09 18:34:06.063570 | controller | - rook_cleanup 2026-03-09 18:34:06.063574 | controller | - lvms 2026-03-09 18:34:06.063579 | controller | - nmstate 2026-03-09 18:34:06.063583 | controller | - nncp 2026-03-09 18:34:06.063587 | controller | - nncp_cleanup 2026-03-09 18:34:06.063591 | controller | - netattach 2026-03-09 18:34:06.063596 | controller | - netattach_cleanup 2026-03-09 18:34:06.063600 | controller | - metallb 2026-03-09 18:34:06.063604 | controller | - metallb_config 2026-03-09 18:34:06.063608 | controller | - metallb_config_cleanup 2026-03-09 18:34:06.063614 | controller | - metallb_cleanup 2026-03-09 18:34:06.063619 | controller | - loki 2026-03-09 18:34:06.063623 | controller | - loki_cleanup 2026-03-09 18:34:06.063627 | controller | - loki_deploy 2026-03-09 18:34:06.063632 | controller | - loki_deploy_cleanup 2026-03-09 18:34:06.063636 | controller | - netobserv 2026-03-09 18:34:06.063640 | controller | - netobserv_cleanup 2026-03-09 18:34:06.063645 | controller | - netobserv_deploy 2026-03-09 18:34:06.063649 | controller | - netobserv_deploy_cleanup 2026-03-09 18:34:06.063653 | controller | - manila_prep 2026-03-09 18:34:06.063658 | controller | - manila 2026-03-09 18:34:06.063662 | controller | - manila_cleanup 2026-03-09 18:34:06.063666 | controller | - manila_deploy_prep 2026-03-09 18:34:06.063671 | controller | - manila_deploy 2026-03-09 18:34:06.063675 | controller | - manila_deploy_cleanup 2026-03-09 18:34:06.063679 | controller | - telemetry_prep 2026-03-09 18:34:06.063684 | controller | - telemetry 2026-03-09 18:34:06.063688 | controller | - telemetry_cleanup 2026-03-09 18:34:06.063692 | controller | - telemetry_deploy_prep 2026-03-09 18:34:06.063696 | controller | - telemetry_deploy 2026-03-09 18:34:06.063701 | controller | - telemetry_deploy_cleanup 2026-03-09 18:34:06.063705 | controller | - telemetry_kuttl_run 2026-03-09 18:34:06.063709 | controller | - telemetry_kuttl 2026-03-09 18:34:06.063713 | controller | - swift_prep 2026-03-09 18:34:06.063719 | controller | - swift 2026-03-09 18:34:06.063723 | controller | - swift_cleanup 2026-03-09 18:34:06.063727 | controller | - swift_deploy_prep 2026-03-09 18:34:06.063732 | controller | - swift_deploy 2026-03-09 18:34:06.063736 | controller | - swift_deploy_cleanup 2026-03-09 18:34:06.063740 | controller | - certmanager 2026-03-09 18:34:06.063744 | controller | - certmanager_cleanup 2026-03-09 18:34:06.063749 | controller | - validate_marketplace 2026-03-09 18:34:06.063753 | controller | - redis_deploy_prep 2026-03-09 18:34:06.063757 | controller | - redis_deploy 2026-03-09 18:34:06.063761 | controller | - redis_deploy_cleanup 2026-03-09 18:34:06.063768 | controller | - set_slower_etcd_profile 2026-03-09 18:34:06.063772 | controller | /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/Makefile: 2026-03-09 18:34:06.063777 | controller | - help 2026-03-09 18:34:06.063781 | controller | - download_tools 2026-03-09 18:34:06.063785 | controller | - nfs 2026-03-09 18:34:06.063789 | controller | - nfs_cleanup 2026-03-09 18:34:06.063794 | controller | - crc 2026-03-09 18:34:06.063798 | controller | - crc_cleanup 2026-03-09 18:34:06.063802 | controller | - crc_scrub 2026-03-09 18:34:06.063806 | controller | - crc_attach_default_interface 2026-03-09 18:34:06.063811 | controller | - crc_attach_default_interface_cleanup 2026-03-09 18:34:06.063815 | controller | - ipv6_lab_network 2026-03-09 18:34:06.063819 | controller | - ipv6_lab_network_cleanup 2026-03-09 18:34:06.063823 | controller | - ipv6_lab_nat64_router 2026-03-09 18:34:06.063828 | controller | - ipv6_lab_nat64_router_cleanup 2026-03-09 18:34:06.063832 | controller | - ipv6_lab_sno 2026-03-09 18:34:06.063836 | controller | - ipv6_lab_sno_cleanup 2026-03-09 18:34:06.063840 | controller | - ipv6_lab 2026-03-09 18:34:06.063845 | controller | - ipv6_lab_cleanup 2026-03-09 18:34:06.063849 | controller | - attach_default_interface 2026-03-09 18:34:06.063853 | controller | - attach_default_interface_cleanup 2026-03-09 18:34:06.063857 | controller | - network_isolation_bridge 2026-03-09 18:34:06.063862 | controller | - network_isolation_bridge_cleanup 2026-03-09 18:34:06.063866 | controller | - edpm_baremetal_compute 2026-03-09 18:34:06.063870 | controller | - edpm_compute 2026-03-09 18:34:06.063876 | controller | - edpm_compute_bootc 2026-03-09 18:34:06.063880 | controller | - edpm_ansible_runner 2026-03-09 18:34:06.063884 | controller | - edpm_computes_bgp 2026-03-09 18:34:06.063889 | controller | - edpm_compute_repos 2026-03-09 18:34:06.063893 | controller | - edpm_compute_cleanup 2026-03-09 18:34:06.063897 | controller | - edpm_networker 2026-03-09 18:34:06.063901 | controller | - edpm_networker_cleanup 2026-03-09 18:34:06.063905 | controller | - edpm_deploy_instance 2026-03-09 18:34:06.063910 | controller | - tripleo_deploy 2026-03-09 18:34:06.063914 | controller | - standalone_deploy 2026-03-09 18:34:06.063918 | controller | - standalone_sync 2026-03-09 18:34:06.063922 | controller | - standalone 2026-03-09 18:34:06.063926 | controller | - standalone_cleanup 2026-03-09 18:34:06.063931 | controller | - standalone_snapshot 2026-03-09 18:34:06.063935 | controller | - standalone_revert 2026-03-09 18:34:06.063939 | controller | - cifmw_prepare 2026-03-09 18:34:06.063943 | controller | - cifmw_cleanup 2026-03-09 18:34:06.063947 | controller | - bmaas_network 2026-03-09 18:34:06.063952 | controller | - bmaas_network_cleanup 2026-03-09 18:34:06.063956 | controller | - bmaas_route_crc_and_crc_bmaas_networks 2026-03-09 18:34:06.063960 | controller | - bmaas_route_crc_and_crc_bmaas_networks_cleanup 2026-03-09 18:34:06.063968 | controller | - bmaas_crc_attach_network 2026-03-09 18:34:06.073436 | controller | - bmaas_crc_attach_network_cleanup 2026-03-09 18:34:06.073456 | controller | - bmaas_crc_baremetal_bridge 2026-03-09 18:34:06.073461 | controller | - bmaas_crc_baremetal_bridge_cleanup 2026-03-09 18:34:06.073466 | controller | - bmaas_baremetal_net_nad 2026-03-09 18:34:06.073470 | controller | - bmaas_baremetal_net_nad_cleanup 2026-03-09 18:34:06.073475 | controller | - bmaas_metallb 2026-03-09 18:34:06.073479 | controller | - bmaas_metallb_cleanup 2026-03-09 18:34:06.073483 | controller | - bmaas_virtual_bms 2026-03-09 18:34:06.073488 | controller | - bmaas_virtual_bms_cleanup 2026-03-09 18:34:06.073492 | controller | - bmaas_sushy_emulator 2026-03-09 18:34:06.073496 | controller | - bmaas_sushy_emulator_cleanup 2026-03-09 18:34:06.073501 | controller | - bmaas_sushy_emulator_wait 2026-03-09 18:34:06.073505 | controller | - bmaas_generate_nodes_yaml 2026-03-09 18:34:06.073509 | controller | - bmaas 2026-03-09 18:34:06.073513 | controller | - bmaas_cleanup 2026-03-09 18:34:06.073518 | controller | failed: false 2026-03-09 18:34:06.073522 | controller | success: true 2026-03-09 18:34:06.073530 | controller | 2026-03-09 18:34:07.008390 | controller | TASK [install_yamls : Create the install_yamls parameters file dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, content={{ 2026-03-09 18:34:07.008424 | controller | { 2026-03-09 18:34:07.008431 | controller | 'cifmw_install_yamls_environment': cifmw_install_yamls_environment, 2026-03-09 18:34:07.008436 | controller | 'cifmw_install_yamls_defaults': cifmw_install_yamls_defaults 2026-03-09 18:34:07.008441 | controller | } | to_nice_yaml 2026-03-09 18:34:07.008446 | controller | }}, mode=0644] *** 2026-03-09 18:34:07.008450 | controller | Monday 09 March 2026 18:34:06 +0000 (0:00:00.028) 0:01:02.008 ********** 2026-03-09 18:34:07.008455 | controller | Monday 09 March 2026 18:34:06 +0000 (0:00:00.028) 0:01:02.006 ********** 2026-03-09 18:34:07.008464 | controller | changed: [controller] 2026-03-09 18:34:07.037631 | controller | 2026-03-09 18:34:07.037662 | controller | TASK [install_yamls : Create empty cifmw_install_yamls_environment if needed cifmw_install_yamls_environment={}] *** 2026-03-09 18:34:07.037669 | controller | Monday 09 March 2026 18:34:07 +0000 (0:00:00.935) 0:01:02.943 ********** 2026-03-09 18:34:07.037674 | controller | Monday 09 March 2026 18:34:07 +0000 (0:00:00.935) 0:01:02.941 ********** 2026-03-09 18:34:07.037682 | controller | skipping: [controller] 2026-03-09 18:34:07.406677 | controller | 2026-03-09 18:34:07.406715 | controller | TASK [discover_latest_image : Get latest image url={{ cifmw_discover_latest_image_base_url }}, image_prefix={{ cifmw_discover_latest_image_qcow_prefix }}, images_file={{ cifmw_discover_latest_image_images_file }}] *** 2026-03-09 18:34:07.406723 | controller | Monday 09 March 2026 18:34:07 +0000 (0:00:00.029) 0:01:02.972 ********** 2026-03-09 18:34:07.406730 | controller | Monday 09 March 2026 18:34:07 +0000 (0:00:00.029) 0:01:02.971 ********** 2026-03-09 18:34:07.406744 | controller | changed: [controller] 2026-03-09 18:34:07.436145 | controller | 2026-03-09 18:34:07.436202 | controller | TASK [discover_latest_image : Export facts accordingly cifmw_discovered_image_name={{ discovered_image['data']['image_name'] }}, cifmw_discovered_image_url={{ discovered_image['data']['image_url'] }}, cifmw_discovered_hash={{ discovered_image['data']['hash'] }}, cifmw_discovered_hash_algorithm={{ discovered_image['data']['hash_algorithm'] }}, cacheable=True] *** 2026-03-09 18:34:07.436217 | controller | Monday 09 March 2026 18:34:07 +0000 (0:00:00.369) 0:01:03.341 ********** 2026-03-09 18:34:07.436229 | controller | Monday 09 March 2026 18:34:07 +0000 (0:00:00.369) 0:01:03.340 ********** 2026-03-09 18:34:07.436241 | controller | ok: [controller] 2026-03-09 18:34:08.105088 | controller | 2026-03-09 18:34:08.105125 | controller | TASK [cifmw_setup : Create artifacts with custom params mode=0644, dest={{ cifmw_basedir }}/artifacts/parameters/custom-params.yml, content={{ ci_framework_params | to_nice_yaml }}] *** 2026-03-09 18:34:08.105133 | controller | Monday 09 March 2026 18:34:07 +0000 (0:00:00.029) 0:01:03.371 ********** 2026-03-09 18:34:08.105139 | controller | Monday 09 March 2026 18:34:07 +0000 (0:00:00.029) 0:01:03.369 ********** 2026-03-09 18:34:08.105270 | controller | changed: [controller] 2026-03-09 18:34:08.135658 | controller | 2026-03-09 18:34:08.135697 | controller | PLAY [Install dev tools] ******************************************************* 2026-03-09 18:34:08.135704 | controller | 2026-03-09 18:34:08.135711 | controller | TASK [Assert that operator_name is set that=['operator_name is defined']] ****** 2026-03-09 18:34:08.135717 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.668) 0:01:04.040 ********** 2026-03-09 18:34:08.135723 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.668) 0:01:04.038 ********** 2026-03-09 18:34:08.135736 | controller | ok: [controller] => 2026-03-09 18:34:08.169532 | controller | changed: false 2026-03-09 18:34:08.169570 | controller | msg: All assertions passed 2026-03-09 18:34:08.169576 | controller | 2026-03-09 18:34:08.169582 | controller | TASK [Download install_yamls deps name=install_yamls_makes, tasks_from=make_download_tools] *** 2026-03-09 18:34:08.169588 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.030) 0:01:04.070 ********** 2026-03-09 18:34:08.169593 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.030) 0:01:04.069 ********** 2026-03-09 18:34:08.169603 | controller | 2026-03-09 18:34:08.192899 | controller | TASK [install_yamls_makes : Debug make_download_tools_env var=make_download_tools_env] *** 2026-03-09 18:34:08.192935 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.033) 0:01:04.104 ********** 2026-03-09 18:34:08.192942 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.033) 0:01:04.103 ********** 2026-03-09 18:34:08.192952 | controller | skipping: [controller] 2026-03-09 18:34:08.217947 | controller | 2026-03-09 18:34:08.217987 | controller | TASK [install_yamls_makes : Debug make_download_tools_params var=make_download_tools_params] *** 2026-03-09 18:34:08.217996 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.023) 0:01:04.127 ********** 2026-03-09 18:34:08.218002 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.023) 0:01:04.126 ********** 2026-03-09 18:34:08.218014 | controller | skipping: [controller] 2026-03-09 18:34:49.145514 | controller | 2026-03-09 18:34:49.145629 | controller | TASK [install_yamls_makes : Run download_tools output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup, script=make download_tools, dry_run={{ make_download_tools_dryrun|default(false)|bool }}, extra_args={{ dict((make_download_tools_env|default({})), **(make_download_tools_params|default({}))) }}] *** 2026-03-09 18:34:49.145636 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.024) 0:01:04.152 ********** 2026-03-09 18:34:49.145642 | controller | Monday 09 March 2026 18:34:08 +0000 (0:00:00.024) 0:01:04.151 ********** 2026-03-09 18:34:49.145651 | controller | [WARNING]: conditional statements should not include jinja2 templating 2026-03-09 18:34:49.158935 | controller | delimiters such as {{ }} or {% %}. Found: {{ make_download_tools_until | 2026-03-09 18:34:49.158969 | controller | default(true) }} 2026-03-09 18:34:49.158979 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_000_run_download.log 2026-03-09 18:34:49.226550 | controller | changed: [controller] 2026-03-09 18:34:49.226585 | controller | 2026-03-09 18:34:49.226592 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-09 18:34:49.226598 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:40.941) 0:01:45.093 ********** 2026-03-09 18:34:49.226603 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:40.941) 0:01:45.092 ********** 2026-03-09 18:34:49.226612 | controller | ok: [controller] 2026-03-09 18:34:49.329718 | controller | 2026-03-09 18:34:49.329752 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-09 18:34:49.329759 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.067) 0:01:45.161 ********** 2026-03-09 18:34:49.329764 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.067) 0:01:45.160 ********** 2026-03-09 18:34:49.329776 | controller | ok: [controller] 2026-03-09 18:34:49.454955 | controller | 2026-03-09 18:34:49.454988 | controller | TASK [run_hook : Loop on hooks for pre_infra _raw_params={{ hook.type }}.yml] *** 2026-03-09 18:34:49.454994 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.103) 0:01:45.264 ********** 2026-03-09 18:34:49.454999 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.103) 0:01:45.263 ********** 2026-03-09 18:34:49.455009 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Download needed tools', 'inventory': 'localhost,', 'connection': 'local', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/download_tools.yaml'}) 2026-03-09 18:34:49.496900 | controller | 2026-03-09 18:34:49.496936 | controller | TASK [run_hook : Set playbook path for Download needed tools cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] *** 2026-03-09 18:34:49.496943 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.125) 0:01:45.389 ********** 2026-03-09 18:34:49.496948 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.125) 0:01:45.388 ********** 2026-03-09 18:34:49.496957 | controller | ok: [controller] 2026-03-09 18:34:49.705928 | controller | 2026-03-09 18:34:49.705964 | controller | TASK [run_hook : Get file stat path={{ playbook_path }}] *********************** 2026-03-09 18:34:49.705971 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.041) 0:01:45.431 ********** 2026-03-09 18:34:49.705976 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.041) 0:01:45.430 ********** 2026-03-09 18:34:49.705987 | controller | ok: [controller] 2026-03-09 18:34:49.726558 | controller | 2026-03-09 18:34:49.726594 | controller | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] *** 2026-03-09 18:34:49.726601 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.208) 0:01:45.640 ********** 2026-03-09 18:34:49.726606 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.208) 0:01:45.639 ********** 2026-03-09 18:34:49.726615 | controller | skipping: [controller] 2026-03-09 18:34:49.934822 | controller | 2026-03-09 18:34:49.934873 | controller | TASK [run_hook : Get parameters files paths={{ 2026-03-09 18:34:49.934883 | controller | (cifmw_basedir, 'artifacts/parameters') | path_join 2026-03-09 18:34:49.934890 | controller | }}, file_type=file, patterns=*.yml] *** 2026-03-09 18:34:49.934898 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.020) 0:01:45.661 ********** 2026-03-09 18:34:49.934906 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.020) 0:01:45.660 ********** 2026-03-09 18:34:49.934928 | controller | ok: [controller] 2026-03-09 18:34:49.966473 | controller | 2026-03-09 18:34:49.966592 | controller | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] *** 2026-03-09 18:34:49.966604 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.208) 0:01:45.869 ********** 2026-03-09 18:34:49.966612 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.208) 0:01:45.868 ********** 2026-03-09 18:34:49.966636 | controller | ok: [controller] 2026-03-09 18:34:50.198905 | controller | 2026-03-09 18:34:50.198945 | controller | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] *** 2026-03-09 18:34:50.198953 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.031) 0:01:45.901 ********** 2026-03-09 18:34:50.198959 | controller | Monday 09 March 2026 18:34:49 +0000 (0:00:00.031) 0:01:45.899 ********** 2026-03-09 18:34:50.198970 | controller | ok: [controller] 2026-03-09 18:34:50.409015 | controller | 2026-03-09 18:34:50.409048 | controller | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-09 18:34:50.409055 | controller | Monday 09 March 2026 18:34:50 +0000 (0:00:00.232) 0:01:46.133 ********** 2026-03-09 18:34:50.409060 | controller | Monday 09 March 2026 18:34:50 +0000 (0:00:00.232) 0:01:46.132 ********** 2026-03-09 18:34:50.409069 | controller | ok: [controller] 2026-03-09 18:35:26.193365 | controller | 2026-03-09 18:35:26.193448 | controller | TASK [run_hook : Run hook without retry - Download needed tools] *************** 2026-03-09 18:35:26.193457 | controller | Monday 09 March 2026 18:34:50 +0000 (0:00:00.210) 0:01:46.344 ********** 2026-03-09 18:35:26.193463 | controller | Monday 09 March 2026 18:34:50 +0000 (0:00:00.210) 0:01:46.342 ********** 2026-03-09 18:35:26.193482 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_001_run_hook_without_retry.log 2026-03-09 18:35:26.217788 | controller | changed: [controller] 2026-03-09 18:35:26.217855 | controller | 2026-03-09 18:35:26.217861 | controller | TASK [run_hook : Run hook with retry - Download needed tools] ****************** 2026-03-09 18:35:26.217866 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:35.783) 0:02:22.127 ********** 2026-03-09 18:35:26.217871 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:35.783) 0:02:22.126 ********** 2026-03-09 18:35:26.217886 | controller | skipping: [controller] 2026-03-09 18:35:26.416886 | controller | 2026-03-09 18:35:26.416927 | controller | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-09 18:35:26.416935 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.024) 0:02:22.152 ********** 2026-03-09 18:35:26.416942 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.024) 0:02:22.151 ********** 2026-03-09 18:35:26.416954 | controller | ok: [controller] 2026-03-09 18:35:26.480461 | controller | 2026-03-09 18:35:26.480496 | controller | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-09 18:35:26.480503 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.199) 0:02:22.351 ********** 2026-03-09 18:35:26.480508 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.199) 0:02:22.350 ********** 2026-03-09 18:35:26.480517 | controller | skipping: [controller] 2026-03-09 18:35:26.525210 | controller | 2026-03-09 18:35:26.525245 | controller | PLAY [Prepare host virtualization] ********************************************* 2026-03-09 18:35:26.525251 | controller | 2026-03-09 18:35:26.525256 | controller | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] *** 2026-03-09 18:35:26.525261 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.063) 0:02:22.415 ********** 2026-03-09 18:35:26.525268 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.063) 0:02:22.414 ********** 2026-03-09 18:35:26.525277 | controller | ok: [controller] 2026-03-09 18:35:26.548575 | controller | 2026-03-09 18:35:26.548611 | controller | TASK [Ensure libvirt is present/configured name=libvirt_manager] *************** 2026-03-09 18:35:26.548617 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.044) 0:02:22.460 ********** 2026-03-09 18:35:26.548624 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.044) 0:02:22.458 ********** 2026-03-09 18:35:26.548634 | controller | skipping: [controller] 2026-03-09 18:35:26.625065 | controller | 2026-03-09 18:35:26.625098 | controller | TASK [Perpare OpenShift provisioner node name=openshift_provisioner_node] ****** 2026-03-09 18:35:26.625105 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.023) 0:02:22.483 ********** 2026-03-09 18:35:26.625110 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.023) 0:02:22.482 ********** 2026-03-09 18:35:26.625118 | controller | skipping: [controller] 2026-03-09 18:35:26.702433 | controller | 2026-03-09 18:35:26.702473 | controller | PLAY [Build dataset hook] ****************************************************** 2026-03-09 18:35:26.702481 | controller | 2026-03-09 18:35:26.702487 | controller | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] *** 2026-03-09 18:35:26.702493 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.076) 0:02:22.560 ********** 2026-03-09 18:35:26.702499 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.076) 0:02:22.558 ********** 2026-03-09 18:35:26.702510 | controller | ok: [controller] 2026-03-09 18:35:26.898202 | controller | 2026-03-09 18:35:26.898265 | controller | TASK [networking_mapper : Check for Networking Environment Definition file existence path={{ cifmw_networking_mapper_networking_env_def_path }}] *** 2026-03-09 18:35:26.898272 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.077) 0:02:22.637 ********** 2026-03-09 18:35:26.898277 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.077) 0:02:22.635 ********** 2026-03-09 18:35:26.898291 | controller | ok: [controller] 2026-03-09 18:35:26.925813 | controller | 2026-03-09 18:35:26.925887 | controller | TASK [networking_mapper : Check for Networking Definition file existance that=['_net_env_def_stat.stat.exists'], msg=Ensure that the Networking Environment Definition file exists in {{ cifmw_networking_mapper_networking_env_def_path }}, quiet=True] *** 2026-03-09 18:35:26.925898 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.195) 0:02:22.833 ********** 2026-03-09 18:35:26.925905 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.195) 0:02:22.831 ********** 2026-03-09 18:35:26.925923 | controller | skipping: [controller] 2026-03-09 18:35:26.950917 | controller | 2026-03-09 18:35:26.950963 | controller | TASK [networking_mapper : Load the Networking Definition from file path={{ cifmw_networking_mapper_networking_env_def_path }}] *** 2026-03-09 18:35:26.950971 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.027) 0:02:22.860 ********** 2026-03-09 18:35:26.950977 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.027) 0:02:22.859 ********** 2026-03-09 18:35:26.950990 | controller | skipping: [controller] 2026-03-09 18:35:26.979199 | controller | 2026-03-09 18:35:26.979253 | controller | TASK [networking_mapper : Set cifmw_networking_env_definition is present cifmw_networking_env_definition={{ 2026-03-09 18:35:26.979259 | controller | _net_env_def_slurp['content'] | 2026-03-09 18:35:26.979265 | controller | b64decode | 2026-03-09 18:35:26.979269 | controller | from_yaml 2026-03-09 18:35:26.979274 | controller | }}, cacheable=True] *** 2026-03-09 18:35:26.979279 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.025) 0:02:22.885 ********** 2026-03-09 18:35:26.979283 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.025) 0:02:22.884 ********** 2026-03-09 18:35:26.979295 | controller | skipping: [controller] 2026-03-09 18:35:27.002274 | controller | 2026-03-09 18:35:27.002326 | controller | TASK [Deploy OCP using Hive name=hive] ***************************************** 2026-03-09 18:35:27.002332 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.028) 0:02:22.914 ********** 2026-03-09 18:35:27.002337 | controller | Monday 09 March 2026 18:35:26 +0000 (0:00:00.028) 0:02:22.912 ********** 2026-03-09 18:35:27.002350 | controller | skipping: [controller] 2026-03-09 18:35:27.026477 | controller | 2026-03-09 18:35:27.026530 | controller | TASK [Prepare CRC name=rhol_crc] *********************************************** 2026-03-09 18:35:27.026537 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.022) 0:02:22.937 ********** 2026-03-09 18:35:27.026541 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.022) 0:02:22.935 ********** 2026-03-09 18:35:27.026554 | controller | skipping: [controller] 2026-03-09 18:35:27.049877 | controller | 2026-03-09 18:35:27.049945 | controller | TASK [Deploy OpenShift cluster using dev-scripts name=devscripts] ************** 2026-03-09 18:35:27.049955 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.024) 0:02:22.961 ********** 2026-03-09 18:35:27.049961 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.024) 0:02:22.960 ********** 2026-03-09 18:35:27.049978 | controller | skipping: [controller] 2026-03-09 18:35:27.258039 | controller | 2026-03-09 18:35:27.258092 | controller | TASK [openshift_login : Ensure output directory exists path={{ cifmw_openshift_login_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-09 18:35:27.258100 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.023) 0:02:22.984 ********** 2026-03-09 18:35:27.258105 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.023) 0:02:22.983 ********** 2026-03-09 18:35:27.258116 | controller | ok: [controller] 2026-03-09 18:35:27.295542 | controller | 2026-03-09 18:35:27.295608 | controller | TASK [openshift_login : OpenShift login _raw_params=login.yml] ***************** 2026-03-09 18:35:27.295622 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.208) 0:02:23.192 ********** 2026-03-09 18:35:27.295626 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.208) 0:02:23.191 ********** 2026-03-09 18:35:27.295640 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/login.yml for controller 2026-03-09 18:35:27.320812 | controller | 2026-03-09 18:35:27.320872 | controller | TASK [openshift_login : Check if the password file is present path={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] *** 2026-03-09 18:35:27.320890 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.037) 0:02:23.230 ********** 2026-03-09 18:35:27.320898 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.037) 0:02:23.229 ********** 2026-03-09 18:35:27.320911 | controller | skipping: [controller] 2026-03-09 18:35:27.346196 | controller | 2026-03-09 18:35:27.346253 | controller | TASK [openshift_login : Fetch user password content src={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] *** 2026-03-09 18:35:27.346261 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.025) 0:02:23.255 ********** 2026-03-09 18:35:27.346266 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.025) 0:02:23.254 ********** 2026-03-09 18:35:27.346278 | controller | skipping: [controller] 2026-03-09 18:35:27.372323 | controller | 2026-03-09 18:35:27.372382 | controller | TASK [openshift_login : Set user password as a fact cifmw_openshift_login_password={{ cifmw_openshift_login_password_file_slurp.content | b64decode }}, cacheable=True] *** 2026-03-09 18:35:27.372389 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.025) 0:02:23.281 ********** 2026-03-09 18:35:27.372394 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.025) 0:02:23.279 ********** 2026-03-09 18:35:27.372408 | controller | skipping: [controller] 2026-03-09 18:35:27.408032 | controller | 2026-03-09 18:35:27.408105 | controller | TASK [openshift_login : Set role variables cifmw_openshift_login_kubeconfig={{ 2026-03-09 18:35:27.408113 | controller | cifmw_openshift_login_kubeconfig | 2026-03-09 18:35:27.408118 | controller | default(cifmw_openshift_kubeconfig) | 2026-03-09 18:35:27.408122 | controller | default( 2026-03-09 18:35:27.408127 | controller | ansible_env.KUBECONFIG if 'KUBECONFIG' in ansible_env else 2026-03-09 18:35:27.408132 | controller | cifmw_openshift_login_kubeconfig_default_path 2026-03-09 18:35:27.408137 | controller | ) | trim 2026-03-09 18:35:27.408142 | controller | }}, cifmw_openshift_login_user={{ cifmw_openshift_login_user | default(cifmw_openshift_user) | default(omit) }}, cifmw_openshift_login_password={{ cifmw_openshift_login_password | default(cifmw_openshift_password) | default(omit) }}, cifmw_openshift_login_api={{ cifmw_openshift_login_api | default(cifmw_openshift_api) | default(omit) }}, cifmw_openshift_login_cert_login={{ cifmw_openshift_login_cert_login | default(false)}}, cifmw_openshift_login_provided_token={{ cifmw_openshift_provided_token | default(omit) }}, cacheable=True] *** 2026-03-09 18:35:27.408148 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.026) 0:02:23.307 ********** 2026-03-09 18:35:27.408184 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.026) 0:02:23.305 ********** 2026-03-09 18:35:27.408199 | controller | ok: [controller] 2026-03-09 18:35:27.619436 | controller | 2026-03-09 18:35:27.619477 | controller | TASK [openshift_login : Check if kubeconfig exists path={{ cifmw_openshift_login_kubeconfig }}] *** 2026-03-09 18:35:27.619485 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.035) 0:02:23.342 ********** 2026-03-09 18:35:27.619491 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.035) 0:02:23.341 ********** 2026-03-09 18:35:27.619502 | controller | ok: [controller] 2026-03-09 18:35:27.647200 | controller | 2026-03-09 18:35:27.647238 | controller | TASK [openshift_login : Assert that enough data is provided to log in to OpenShift that=cifmw_openshift_login_kubeconfig_stat.stat.exists or (cifmw_openshift_login_provided_token is defined and cifmw_openshift_login_provided_token != '') or ( 2026-03-09 18:35:27.647246 | controller | (cifmw_openshift_login_user is defined) and 2026-03-09 18:35:27.647253 | controller | (cifmw_openshift_login_password is defined) and 2026-03-09 18:35:27.647258 | controller | (cifmw_openshift_login_api is defined) 2026-03-09 18:35:27.647265 | controller | ), msg=If an existing kubeconfig is not provided user/pwd or provided/initial token and API URL must be given] *** 2026-03-09 18:35:27.647270 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.210) 0:02:23.553 ********** 2026-03-09 18:35:27.647276 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.211) 0:02:23.552 ********** 2026-03-09 18:35:27.647287 | controller | ok: [controller] => 2026-03-09 18:35:27.676591 | controller | changed: false 2026-03-09 18:35:27.676632 | controller | msg: All assertions passed 2026-03-09 18:35:27.676641 | controller | 2026-03-09 18:35:27.676648 | controller | TASK [openshift_login : Fetch kubeconfig content src={{ cifmw_openshift_login_kubeconfig }}] *** 2026-03-09 18:35:27.676656 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.028) 0:02:23.582 ********** 2026-03-09 18:35:27.676662 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.028) 0:02:23.580 ********** 2026-03-09 18:35:27.676673 | controller | skipping: [controller] 2026-03-09 18:35:27.699529 | controller | 2026-03-09 18:35:27.699562 | controller | TASK [openshift_login : Fetch x509 key based users cifmw_openshift_login_key_based_users={{ 2026-03-09 18:35:27.699569 | controller | ( 2026-03-09 18:35:27.699574 | controller | cifmw_openshift_login_kubeconfig_content_b64.content | 2026-03-09 18:35:27.699579 | controller | b64decode | 2026-03-09 18:35:27.699583 | controller | from_yaml 2026-03-09 18:35:27.699588 | controller | ). users | default([]) | 2026-03-09 18:35:27.699595 | controller | selectattr('user.client-certificate-data', 'defined') | 2026-03-09 18:35:27.699600 | controller | map(attribute="name") | 2026-03-09 18:35:27.699604 | controller | map("split", "/") | 2026-03-09 18:35:27.699609 | controller | map("first") 2026-03-09 18:35:27.699613 | controller | }}, cacheable=True] *** 2026-03-09 18:35:27.699618 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.029) 0:02:23.611 ********** 2026-03-09 18:35:27.699622 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.029) 0:02:23.610 ********** 2026-03-09 18:35:27.699631 | controller | skipping: [controller] 2026-03-09 18:35:27.720583 | controller | 2026-03-09 18:35:27.720616 | controller | TASK [openshift_login : Assign key based user if not provided and available cifmw_openshift_login_user={{ 2026-03-09 18:35:27.720623 | controller | (cifmw_openshift_login_assume_cert_system_user | ternary('system:', '')) + 2026-03-09 18:35:27.720628 | controller | (cifmw_openshift_login_key_based_users | map('replace', 'system:', '') | unique | first) 2026-03-09 18:35:27.720633 | controller | }}, cifmw_openshift_login_cert_login=True, cacheable=True] *** 2026-03-09 18:35:27.720638 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.023) 0:02:23.634 ********** 2026-03-09 18:35:27.720643 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.023) 0:02:23.633 ********** 2026-03-09 18:35:27.720652 | controller | skipping: [controller] 2026-03-09 18:35:27.752279 | controller | 2026-03-09 18:35:27.752319 | controller | TASK [openshift_login : Set the retry count cifmw_openshift_login_retries_cnt={{ 2026-03-09 18:35:27.752328 | controller | 0 if cifmw_openshift_login_retries_cnt is undefined else 2026-03-09 18:35:27.752334 | controller | cifmw_openshift_login_retries_cnt|int + 1 2026-03-09 18:35:27.752346 | controller | }}] *** 2026-03-09 18:35:27.752353 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.021) 0:02:23.655 ********** 2026-03-09 18:35:27.752359 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.021) 0:02:23.654 ********** 2026-03-09 18:35:27.752371 | controller | ok: [controller] 2026-03-09 18:35:27.781249 | controller | 2026-03-09 18:35:27.781288 | controller | TASK [openshift_login : Fetch token _raw_params=try_login.yml] ***************** 2026-03-09 18:35:27.781297 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.031) 0:02:23.687 ********** 2026-03-09 18:35:27.781303 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.031) 0:02:23.685 ********** 2026-03-09 18:35:27.781316 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/try_login.yml for controller 2026-03-09 18:35:27.802338 | controller | 2026-03-09 18:35:27.802373 | controller | TASK [openshift_login : Try get OpenShift access token _raw_params=oc whoami -t] *** 2026-03-09 18:35:27.802380 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.028) 0:02:23.716 ********** 2026-03-09 18:35:27.802385 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.028) 0:02:23.714 ********** 2026-03-09 18:35:27.802396 | controller | skipping: [controller] 2026-03-09 18:35:28.408678 | controller | 2026-03-09 18:35:28.408717 | controller | TASK [openshift_login : Fetch OpenShift token output_dir={{ cifmw_openshift_login_basedir }}/artifacts, script=oc login {%- if cifmw_openshift_login_provided_token is not defined %} {%- if cifmw_openshift_login_user is defined %} -u {{ cifmw_openshift_login_user }} {%- endif %} {%- if cifmw_openshift_login_password is defined %} -p {{ cifmw_openshift_login_password }} {%- endif %} {% else %} --token={{ cifmw_openshift_login_provided_token }} {%- endif %} {%- if cifmw_openshift_login_skip_tls_verify|bool %} --insecure-skip-tls-verify=true {%- endif %} {%- if cifmw_openshift_login_api is defined %} {{ cifmw_openshift_login_api }} {%- endif %}] *** 2026-03-09 18:35:28.408727 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.021) 0:02:23.737 ********** 2026-03-09 18:35:28.408734 | controller | Monday 09 March 2026 18:35:27 +0000 (0:00:00.021) 0:02:23.735 ********** 2026-03-09 18:35:28.408746 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_002_fetch_openshift.log 2026-03-09 18:35:28.434409 | controller | changed: [controller] 2026-03-09 18:35:28.434444 | controller | 2026-03-09 18:35:28.434450 | controller | TASK [openshift_login : Ensure kubeconfig is provided that=cifmw_openshift_login_kubeconfig != ""] *** 2026-03-09 18:35:28.434456 | controller | Monday 09 March 2026 18:35:28 +0000 (0:00:00.606) 0:02:24.343 ********** 2026-03-09 18:35:28.434461 | controller | Monday 09 March 2026 18:35:28 +0000 (0:00:00.606) 0:02:24.342 ********** 2026-03-09 18:35:28.434470 | controller | ok: [controller] => 2026-03-09 18:35:28.736440 | controller | changed: false 2026-03-09 18:35:28.736481 | controller | msg: All assertions passed 2026-03-09 18:35:28.736490 | controller | 2026-03-09 18:35:28.736497 | controller | TASK [openshift_login : Fetch new OpenShift access token _raw_params=oc whoami -t] *** 2026-03-09 18:35:28.736503 | controller | Monday 09 March 2026 18:35:28 +0000 (0:00:00.025) 0:02:24.369 ********** 2026-03-09 18:35:28.736510 | controller | Monday 09 March 2026 18:35:28 +0000 (0:00:00.025) 0:02:24.368 ********** 2026-03-09 18:35:28.736521 | controller | changed: [controller] 2026-03-09 18:35:28.768266 | controller | 2026-03-09 18:35:28.768299 | controller | TASK [openshift_login : Set new OpenShift token cifmw_openshift_login_token={{ 2026-03-09 18:35:28.768305 | controller | (not cifmw_openshift_login_new_token_out.skipped | default(false)) | 2026-03-09 18:35:28.768310 | controller | ternary(cifmw_openshift_login_new_token_out.stdout, cifmw_openshift_login_whoami_out.stdout) 2026-03-09 18:35:28.768315 | controller | }}, cacheable=True] *** 2026-03-09 18:35:28.768320 | controller | Monday 09 March 2026 18:35:28 +0000 (0:00:00.301) 0:02:24.671 ********** 2026-03-09 18:35:28.768325 | controller | Monday 09 March 2026 18:35:28 +0000 (0:00:00.301) 0:02:24.669 ********** 2026-03-09 18:35:28.768333 | controller | ok: [controller] 2026-03-09 18:35:29.061208 | controller | 2026-03-09 18:35:29.061248 | controller | TASK [openshift_login : Fetch OpenShift API URL _raw_params=oc whoami --show-server=true] *** 2026-03-09 18:35:29.061263 | controller | Monday 09 March 2026 18:35:28 +0000 (0:00:00.032) 0:02:24.703 ********** 2026-03-09 18:35:29.061270 | controller | Monday 09 March 2026 18:35:28 +0000 (0:00:00.031) 0:02:24.701 ********** 2026-03-09 18:35:29.061282 | controller | changed: [controller] 2026-03-09 18:35:29.385037 | controller | 2026-03-09 18:35:29.385078 | controller | TASK [openshift_login : Fetch OpenShift kubeconfig context _raw_params=oc whoami -c] *** 2026-03-09 18:35:29.385087 | controller | Monday 09 March 2026 18:35:29 +0000 (0:00:00.292) 0:02:24.996 ********** 2026-03-09 18:35:29.385094 | controller | Monday 09 March 2026 18:35:29 +0000 (0:00:00.292) 0:02:24.994 ********** 2026-03-09 18:35:29.385107 | controller | changed: [controller] 2026-03-09 18:35:29.737414 | controller | 2026-03-09 18:35:29.737452 | controller | TASK [openshift_login : Fetch OpenShift current user _raw_params=oc whoami] **** 2026-03-09 18:35:29.737460 | controller | Monday 09 March 2026 18:35:29 +0000 (0:00:00.323) 0:02:25.319 ********** 2026-03-09 18:35:29.737466 | controller | Monday 09 March 2026 18:35:29 +0000 (0:00:00.323) 0:02:25.318 ********** 2026-03-09 18:35:29.737477 | controller | changed: [controller] 2026-03-09 18:35:29.771670 | controller | 2026-03-09 18:35:29.771710 | controller | TASK [openshift_login : Set OpenShift user, context and API facts cifmw_openshift_login_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_login_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_login_user={{ _oauth_user }}, cifmw_openshift_kubeconfig={{ cifmw_openshift_login_kubeconfig }}, cifmw_openshift_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_user={{ _oauth_user }}, cifmw_openshift_token={{ cifmw_openshift_login_token | default(omit) }}, cifmw_install_yamls_environment={{ ( cifmw_install_yamls_environment | 2026-03-09 18:35:29.771725 | controller | combine({'KUBECONFIG': cifmw_openshift_login_kubeconfig}) 2026-03-09 18:35:29.771733 | controller | ) if cifmw_install_yamls_environment is defined else omit 2026-03-09 18:35:29.771739 | controller | }}, cacheable=True] *** 2026-03-09 18:35:29.771746 | controller | Monday 09 March 2026 18:35:29 +0000 (0:00:00.352) 0:02:25.672 ********** 2026-03-09 18:35:29.771752 | controller | Monday 09 March 2026 18:35:29 +0000 (0:00:00.352) 0:02:25.670 ********** 2026-03-09 18:35:29.771763 | controller | ok: [controller] 2026-03-09 18:35:30.298706 | controller | 2026-03-09 18:35:30.298746 | controller | TASK [openshift_login : Create the openshift_login parameters file dest={{ cifmw_basedir }}/artifacts/parameters/openshift-login-params.yml, content={{ cifmw_openshift_login_params_content | from_yaml | to_nice_yaml }}, mode=0600] *** 2026-03-09 18:35:30.298755 | controller | Monday 09 March 2026 18:35:29 +0000 (0:00:00.034) 0:02:25.706 ********** 2026-03-09 18:35:30.298761 | controller | Monday 09 March 2026 18:35:29 +0000 (0:00:00.034) 0:02:25.705 ********** 2026-03-09 18:35:30.298773 | controller | changed: [controller] 2026-03-09 18:35:30.646279 | controller | 2026-03-09 18:35:30.646319 | controller | TASK [openshift_login : Read the install yamls parameters file path={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml] *** 2026-03-09 18:35:30.646328 | controller | Monday 09 March 2026 18:35:30 +0000 (0:00:00.526) 0:02:26.233 ********** 2026-03-09 18:35:30.646334 | controller | Monday 09 March 2026 18:35:30 +0000 (0:00:00.526) 0:02:26.232 ********** 2026-03-09 18:35:30.646345 | controller | ok: [controller] 2026-03-09 18:35:31.311859 | controller | 2026-03-09 18:35:31.311892 | controller | TASK [openshift_login : Append the KUBECONFIG to the install yamls parameters content={{ 2026-03-09 18:35:31.311898 | controller | cifmw_openshift_login_install_yamls_artifacts_slurp['content'] | b64decode | from_yaml | 2026-03-09 18:35:31.311903 | controller | combine( 2026-03-09 18:35:31.311908 | controller | { 2026-03-09 18:35:31.311912 | controller | 'cifmw_install_yamls_environment': { 2026-03-09 18:35:31.311917 | controller | 'KUBECONFIG': cifmw_openshift_login_kubeconfig 2026-03-09 18:35:31.311922 | controller | } 2026-03-09 18:35:31.311926 | controller | }, recursive=true) | to_nice_yaml 2026-03-09 18:35:31.311930 | controller | }}, dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, mode=0600] *** 2026-03-09 18:35:31.311939 | controller | Monday 09 March 2026 18:35:30 +0000 (0:00:00.347) 0:02:26.581 ********** 2026-03-09 18:35:31.311944 | controller | Monday 09 March 2026 18:35:30 +0000 (0:00:00.347) 0:02:26.579 ********** 2026-03-09 18:35:31.311952 | controller | changed: [controller] 2026-03-09 18:35:31.556627 | controller | 2026-03-09 18:35:31.556670 | controller | TASK [openshift_setup : Ensure output directory exists path={{ cifmw_openshift_setup_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-09 18:35:31.556682 | controller | Monday 09 March 2026 18:35:31 +0000 (0:00:00.665) 0:02:27.246 ********** 2026-03-09 18:35:31.556692 | controller | Monday 09 March 2026 18:35:31 +0000 (0:00:00.665) 0:02:27.245 ********** 2026-03-09 18:35:31.556705 | controller | ok: [controller] 2026-03-09 18:35:31.593096 | controller | 2026-03-09 18:35:31.593135 | controller | TASK [openshift_setup : Fetch namespaces to create cifmw_openshift_setup_namespaces={{ 2026-03-09 18:35:31.593144 | controller | (( 2026-03-09 18:35:31.593176 | controller | ([cifmw_install_yamls_defaults['NAMESPACE']] + 2026-03-09 18:35:31.593182 | controller | ([cifmw_install_yamls_defaults['OPERATOR_NAMESPACE']] if 'OPERATOR_NAMESPACE' is in cifmw_install_yamls_defaults else []) 2026-03-09 18:35:31.593188 | controller | ) if cifmw_install_yamls_defaults is defined else [] 2026-03-09 18:35:31.593193 | controller | ) + cifmw_openshift_setup_create_namespaces) | unique 2026-03-09 18:35:31.593198 | controller | }}] *** 2026-03-09 18:35:31.593203 | controller | Monday 09 March 2026 18:35:31 +0000 (0:00:00.244) 0:02:27.491 ********** 2026-03-09 18:35:31.593208 | controller | Monday 09 March 2026 18:35:31 +0000 (0:00:00.244) 0:02:27.490 ********** 2026-03-09 18:35:31.593220 | controller | ok: [controller] 2026-03-09 18:35:33.836409 | controller | 2026-03-09 18:35:33.836450 | controller | TASK [openshift_setup : Create required namespaces kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit) }}, name={{ item }}, kind=Namespace, state=present] *** 2026-03-09 18:35:33.836458 | controller | Monday 09 March 2026 18:35:31 +0000 (0:00:00.036) 0:02:27.527 ********** 2026-03-09 18:35:33.836464 | controller | Monday 09 March 2026 18:35:31 +0000 (0:00:00.036) 0:02:27.526 ********** 2026-03-09 18:35:33.836475 | controller | changed: [controller] => (item=openstack) 2026-03-09 18:35:33.855800 | controller | changed: [controller] => (item=openstack-operators) 2026-03-09 18:35:33.855844 | controller | 2026-03-09 18:35:33.855853 | controller | TASK [openshift_setup : Get internal OpenShift registry route kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, kind=Route, name=default-route, namespace=openshift-image-registry] *** 2026-03-09 18:35:33.855860 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:02.243) 0:02:29.771 ********** 2026-03-09 18:35:33.855866 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:02.243) 0:02:29.769 ********** 2026-03-09 18:35:33.855877 | controller | skipping: [controller] 2026-03-09 18:35:33.881785 | controller | 2026-03-09 18:35:33.881821 | controller | TASK [openshift_setup : Allow anonymous image-pulls in CRC registry for targeted namespaces state=present, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'kind': 'RoleBinding', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'system:image-puller', 'namespace': '{{ item }}'}, 'subjects': [{'kind': 'User', 'name': 'system:anonymous'}, {'kind': 'User', 'name': 'system:unauthenticated'}], 'roleRef': {'kind': 'ClusterRole', 'name': 'system:image-puller'}}] *** 2026-03-09 18:35:33.881829 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.019) 0:02:29.790 ********** 2026-03-09 18:35:33.881834 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.019) 0:02:29.789 ********** 2026-03-09 18:35:33.881843 | controller | skipping: [controller] => (item=openstack) 2026-03-09 18:35:33.909798 | controller | skipping: [controller] => (item=openstack-operators) 2026-03-09 18:35:33.909870 | controller | skipping: [controller] 2026-03-09 18:35:33.909879 | controller | 2026-03-09 18:35:33.909885 | controller | TASK [openshift_setup : Wait for the image registry to be ready kind=Deployment, name=image-registry, namespace=openshift-image-registry, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, wait=True, wait_sleep=10, wait_timeout=600, wait_condition={'type': 'Available', 'status': 'True'}] *** 2026-03-09 18:35:33.909938 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.026) 0:02:29.816 ********** 2026-03-09 18:35:33.909946 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.026) 0:02:29.815 ********** 2026-03-09 18:35:33.909957 | controller | skipping: [controller] 2026-03-09 18:35:33.937866 | controller | 2026-03-09 18:35:33.937904 | controller | TASK [openshift_setup : Login into OpenShift internal registry output_dir={{ cifmw_openshift_setup_basedir }}/artifacts, script=podman login -u {{ cifmw_openshift_user }} -p {{ cifmw_openshift_token }} {%- if cifmw_openshift_setup_skip_internal_registry_tls_verify|bool %} --tls-verify=false {%- endif %} {{ cifmw_openshift_setup_registry_default_route.resources[0].spec.host }}] *** 2026-03-09 18:35:33.937911 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.027) 0:02:29.844 ********** 2026-03-09 18:35:33.937934 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.027) 0:02:29.843 ********** 2026-03-09 18:35:33.937945 | controller | skipping: [controller] 2026-03-09 18:35:33.962991 | controller | 2026-03-09 18:35:33.963029 | controller | TASK [Ensure we have custom CA installed on host role=install_ca] ************** 2026-03-09 18:35:33.963038 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.028) 0:02:29.872 ********** 2026-03-09 18:35:33.963044 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.028) 0:02:29.871 ********** 2026-03-09 18:35:33.963055 | controller | skipping: [controller] 2026-03-09 18:35:33.989279 | controller | 2026-03-09 18:35:33.989322 | controller | TASK [openshift_setup : Update ca bundle _raw_params=update-ca-trust extract] *** 2026-03-09 18:35:33.989330 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.025) 0:02:29.897 ********** 2026-03-09 18:35:33.989337 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.025) 0:02:29.896 ********** 2026-03-09 18:35:33.989351 | controller | skipping: [controller] 2026-03-09 18:35:34.013360 | controller | 2026-03-09 18:35:34.013398 | controller | TASK [openshift_setup : Slurp CAs file src={{ cifmw_openshift_setup_ca_bundle_path }}] *** 2026-03-09 18:35:34.013407 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.026) 0:02:29.924 ********** 2026-03-09 18:35:34.013414 | controller | Monday 09 March 2026 18:35:33 +0000 (0:00:00.026) 0:02:29.922 ********** 2026-03-09 18:35:34.013426 | controller | skipping: [controller] 2026-03-09 18:35:34.038988 | controller | 2026-03-09 18:35:34.039031 | controller | TASK [openshift_setup : Create config map with registry CAs kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'namespace': 'openshift-config', 'name': 'registry-cas'}, 'data': '{{ _config_map_data | items2dict }}'}] *** 2026-03-09 18:35:34.039041 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.024) 0:02:29.948 ********** 2026-03-09 18:35:34.039047 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.024) 0:02:29.946 ********** 2026-03-09 18:35:34.039059 | controller | skipping: [controller] 2026-03-09 18:35:34.066552 | controller | 2026-03-09 18:35:34.066592 | controller | TASK [openshift_setup : Install Red Hat CA for pulling images from internal registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'additionalTrustedCA': {'name': 'registry-cas'}}}] *** 2026-03-09 18:35:34.066604 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.025) 0:02:29.973 ********** 2026-03-09 18:35:34.066611 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.025) 0:02:29.972 ********** 2026-03-09 18:35:34.066623 | controller | skipping: [controller] 2026-03-09 18:35:34.092323 | controller | 2026-03-09 18:35:34.092368 | controller | TASK [openshift_setup : Add insecure registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'registrySources': {'insecureRegistries': ['{{ cifmw_update_containers_registry }}'], 'allowedRegistries': '{{ all_registries }}'}}}] *** 2026-03-09 18:35:34.092379 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.027) 0:02:30.001 ********** 2026-03-09 18:35:34.092386 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.027) 0:02:29.999 ********** 2026-03-09 18:35:34.092399 | controller | skipping: [controller] 2026-03-09 18:35:34.123358 | controller | 2026-03-09 18:35:34.123397 | controller | TASK [openshift_setup : Create a ICSP with repository digest mirrors kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'operator.openshift.io/v1alpha1', 'kind': 'ImageContentSourcePolicy', 'metadata': {'name': 'registry-digest-mirrors'}, 'spec': {'repositoryDigestMirrors': '{{ cifmw_openshift_setup_digest_mirrors }}'}}] *** 2026-03-09 18:35:34.123406 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.025) 0:02:30.027 ********** 2026-03-09 18:35:34.123413 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.025) 0:02:30.025 ********** 2026-03-09 18:35:34.123428 | controller | skipping: [controller] 2026-03-09 18:35:35.181230 | controller | 2026-03-09 18:35:35.181269 | controller | TASK [openshift_setup : Gather network.operator info kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=operator.openshift.io/v1, kind=Network, name=cluster] *** 2026-03-09 18:35:35.181278 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.030) 0:02:30.058 ********** 2026-03-09 18:35:35.181284 | controller | Monday 09 March 2026 18:35:34 +0000 (0:00:00.030) 0:02:30.056 ********** 2026-03-09 18:35:35.181296 | controller | ok: [controller] 2026-03-09 18:35:36.264729 | controller | 2026-03-09 18:35:36.264807 | controller | TASK [openshift_setup : Patch network operator api_version=operator.openshift.io/v1, kubeconfig={{ cifmw_openshift_kubeconfig }}, kind=Network, name=cluster, persist_config=True, patch=[{'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/routingViaHost', 'value': True, 'op': 'replace'}, {'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/ipForwarding', 'value': 'Global', 'op': 'replace'}]] *** 2026-03-09 18:35:36.264823 | controller | Monday 09 March 2026 18:35:35 +0000 (0:00:01.057) 0:02:31.115 ********** 2026-03-09 18:35:36.264830 | controller | Monday 09 March 2026 18:35:35 +0000 (0:00:01.057) 0:02:31.114 ********** 2026-03-09 18:35:36.264849 | controller | changed: [controller] 2026-03-09 18:35:37.151694 | controller | 2026-03-09 18:35:37.151728 | controller | TASK [openshift_setup : Patch samples registry configuration kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=samples.operator.openshift.io/v1, kind=Config, name=cluster, patch=[{'op': 'replace', 'path': '/spec/samplesRegistry', 'value': '{{ cifmw_openshift_setup_samples_registry }}'}]] *** 2026-03-09 18:35:37.151736 | controller | Monday 09 March 2026 18:35:36 +0000 (0:00:01.083) 0:02:32.199 ********** 2026-03-09 18:35:37.151741 | controller | Monday 09 March 2026 18:35:36 +0000 (0:00:01.083) 0:02:32.198 ********** 2026-03-09 18:35:37.151750 | controller | changed: [controller] 2026-03-09 18:35:37.177248 | controller | 2026-03-09 18:35:37.177284 | controller | TASK [openshift_setup : Delete the pods from openshift-marketplace namespace kind=Pod, state=absent, delete_all=True, kubeconfig={{ cifmw_openshift_kubeconfig }}, namespace=openshift-marketplace] *** 2026-03-09 18:35:37.177291 | controller | Monday 09 March 2026 18:35:37 +0000 (0:00:00.887) 0:02:33.086 ********** 2026-03-09 18:35:37.177296 | controller | Monday 09 March 2026 18:35:37 +0000 (0:00:00.887) 0:02:33.085 ********** 2026-03-09 18:35:37.177310 | controller | skipping: [controller] 2026-03-09 18:35:37.210493 | controller | 2026-03-09 18:35:37.210527 | controller | TASK [openshift_setup : Wait for openshift-marketplace pods to be running _raw_params=oc wait pod --all --for=condition=Ready -n openshift-marketplace --timeout=1m] *** 2026-03-09 18:35:37.210535 | controller | Monday 09 March 2026 18:35:37 +0000 (0:00:00.025) 0:02:33.112 ********** 2026-03-09 18:35:37.210539 | controller | Monday 09 March 2026 18:35:37 +0000 (0:00:00.025) 0:02:33.110 ********** 2026-03-09 18:35:37.210548 | controller | skipping: [controller] 2026-03-09 18:35:37.289775 | controller | 2026-03-09 18:35:37.289809 | controller | TASK [Deploy Observability operator. name=openshift_obs] *********************** 2026-03-09 18:35:37.289815 | controller | Monday 09 March 2026 18:35:37 +0000 (0:00:00.033) 0:02:33.145 ********** 2026-03-09 18:35:37.289820 | controller | Monday 09 March 2026 18:35:37 +0000 (0:00:00.033) 0:02:33.144 ********** 2026-03-09 18:35:37.289829 | controller | 2026-03-09 18:35:38.133938 | controller | TASK [openshift_obs : Install cluster observability operator. definition={{cifmw_openshift_obs_definition }}, kubeconfig={{ cifmw_openshift_kubeconfig }}, state=present] *** 2026-03-09 18:35:38.133973 | controller | Monday 09 March 2026 18:35:37 +0000 (0:00:00.079) 0:02:33.224 ********** 2026-03-09 18:35:38.133989 | controller | Monday 09 March 2026 18:35:37 +0000 (0:00:00.079) 0:02:33.223 ********** 2026-03-09 18:35:38.133999 | controller | changed: [controller] 2026-03-09 18:36:24.478731 | controller | 2026-03-09 18:36:24.478813 | controller | TASK [openshift_obs : Wait for observability operator deployment kind=Deployment, namespace=openshift-operators, name=observability-operator, wait=True, wait_timeout=300, wait_condition={'type': 'Available', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] *** 2026-03-09 18:36:24.478820 | controller | Monday 09 March 2026 18:35:38 +0000 (0:00:00.844) 0:02:34.068 ********** 2026-03-09 18:36:24.478826 | controller | Monday 09 March 2026 18:35:38 +0000 (0:00:00.844) 0:02:34.067 ********** 2026-03-09 18:36:24.478842 | controller | ok: [controller] 2026-03-09 18:36:25.455397 | controller | 2026-03-09 18:36:25.455476 | controller | TASK [openshift_obs : Wait for observability-operator pod kind=Pod, namespace=openshift-operators, label_selectors=['app.kubernetes.io/name = observability-operator'], wait=True, wait_timeout=300, wait_condition={'type': 'Ready', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] *** 2026-03-09 18:36:25.455486 | controller | Monday 09 March 2026 18:36:24 +0000 (0:00:46.344) 0:03:20.413 ********** 2026-03-09 18:36:25.455493 | controller | Monday 09 March 2026 18:36:24 +0000 (0:00:46.344) 0:03:20.411 ********** 2026-03-09 18:36:25.455511 | controller | ok: [controller] 2026-03-09 18:36:25.483221 | controller | 2026-03-09 18:36:25.483298 | controller | TASK [Deploy Metal3 BMHs name=deploy_bmh] ************************************** 2026-03-09 18:36:25.483307 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.976) 0:03:21.389 ********** 2026-03-09 18:36:25.483314 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.976) 0:03:21.388 ********** 2026-03-09 18:36:25.483333 | controller | skipping: [controller] 2026-03-09 18:36:25.513245 | controller | 2026-03-09 18:36:25.513314 | controller | TASK [Install certmanager operator role name=cert_manager] ********************* 2026-03-09 18:36:25.513610 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.027) 0:03:21.417 ********** 2026-03-09 18:36:25.513623 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.027) 0:03:21.416 ********** 2026-03-09 18:36:25.513635 | controller | skipping: [controller] 2026-03-09 18:36:25.538213 | controller | 2026-03-09 18:36:25.538248 | controller | TASK [Configure hosts networking using nmstate name=ci_nmstate] **************** 2026-03-09 18:36:25.538257 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.030) 0:03:21.447 ********** 2026-03-09 18:36:25.538263 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.030) 0:03:21.446 ********** 2026-03-09 18:36:25.538275 | controller | skipping: [controller] 2026-03-09 18:36:25.558530 | controller | 2026-03-09 18:36:25.558572 | controller | TASK [Configure multus networks name=ci_multus] ******************************** 2026-03-09 18:36:25.558582 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.025) 0:03:21.473 ********** 2026-03-09 18:36:25.558595 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.025) 0:03:21.471 ********** 2026-03-09 18:36:25.558608 | controller | skipping: [controller] 2026-03-09 18:36:25.581290 | controller | 2026-03-09 18:36:25.581324 | controller | TASK [Deploy Sushy Emulator service pod name=sushy_emulator] ******************* 2026-03-09 18:36:25.581331 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.020) 0:03:21.493 ********** 2026-03-09 18:36:25.581336 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.020) 0:03:21.492 ********** 2026-03-09 18:36:25.581344 | controller | skipping: [controller] 2026-03-09 18:36:25.604379 | controller | 2026-03-09 18:36:25.604414 | controller | TASK [Setup Libvirt on controller name=libvirt_manager] ************************ 2026-03-09 18:36:25.604421 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.022) 0:03:21.516 ********** 2026-03-09 18:36:25.604425 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.022) 0:03:21.514 ********** 2026-03-09 18:36:25.604435 | controller | skipping: [controller] 2026-03-09 18:36:25.630315 | controller | 2026-03-09 18:36:25.630351 | controller | TASK [Prepare container package builder name=pkg_build] ************************ 2026-03-09 18:36:25.630357 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.023) 0:03:21.539 ********** 2026-03-09 18:36:25.630362 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.023) 0:03:21.537 ********** 2026-03-09 18:36:25.630371 | controller | skipping: [controller] 2026-03-09 18:36:25.687858 | controller | 2026-03-09 18:36:25.687893 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-09 18:36:25.687902 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.025) 0:03:21.565 ********** 2026-03-09 18:36:25.687907 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.025) 0:03:21.563 ********** 2026-03-09 18:36:25.687916 | controller | ok: [controller] 2026-03-09 18:36:25.795333 | controller | 2026-03-09 18:36:25.795376 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-09 18:36:25.795386 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.057) 0:03:21.622 ********** 2026-03-09 18:36:25.795393 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.057) 0:03:21.621 ********** 2026-03-09 18:36:25.795405 | controller | ok: [controller] 2026-03-09 18:36:25.910630 | controller | 2026-03-09 18:36:25.910666 | controller | TASK [run_hook : Loop on hooks for post_infra _raw_params={{ hook.type }}.yml] *** 2026-03-09 18:36:25.910673 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.107) 0:03:21.730 ********** 2026-03-09 18:36:25.910678 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.107) 0:03:21.728 ********** 2026-03-09 18:36:25.910687 | controller | skipping: [controller] 2026-03-09 18:36:26.054328 | controller | 2026-03-09 18:36:26.054364 | controller | TASK [Load parameters dir={{ item }}] ****************************************** 2026-03-09 18:36:26.054370 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.115) 0:03:21.845 ********** 2026-03-09 18:36:26.054375 | controller | Monday 09 March 2026 18:36:25 +0000 (0:00:00.115) 0:03:21.844 ********** 2026-03-09 18:36:26.054384 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-09 18:36:26.089488 | controller | ok: [controller] => (item=/etc/ci/env) 2026-03-09 18:36:26.089523 | controller | 2026-03-09 18:36:26.089530 | controller | TASK [Ensure that the isolated net was configured for crc that=['crc_ci_bootstrap_networks_out is defined', 'crc_ci_bootstrap_networks_out[_crc_hostname] is defined', "crc_ci_bootstrap_networks_out[_crc_hostname]['default'] is defined"]] *** 2026-03-09 18:36:26.089538 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.143) 0:03:21.989 ********** 2026-03-09 18:36:26.089543 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.143) 0:03:21.987 ********** 2026-03-09 18:36:26.089552 | controller | ok: [controller] => 2026-03-09 18:36:26.148715 | controller | changed: false 2026-03-09 18:36:26.148755 | controller | msg: All assertions passed 2026-03-09 18:36:26.148761 | controller | 2026-03-09 18:36:26.148767 | controller | TASK [Set facts for further usage within the framework cifmw_edpm_prepare_extra_vars={'NNCP_INTERFACE': '{{ crc_ci_bootstrap_networks_out.crc.default.iface }}', 'NETWORK_MTU': '{{ crc_ci_bootstrap_networks_out.crc.default.mtu }}', 'NNCP_DNS_SERVER': "{{\n cifmw_nncp_dns_server |\n default(crc_ci_bootstrap_networks_out[_crc_hostname].default.ip) |\n split('/') | first\n}}"}] *** 2026-03-09 18:36:26.148772 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.035) 0:03:22.024 ********** 2026-03-09 18:36:26.148777 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.035) 0:03:22.023 ********** 2026-03-09 18:36:26.148786 | controller | ok: [controller] 2026-03-09 18:36:26.200607 | controller | 2026-03-09 18:36:26.200639 | controller | PLAY [Deploy Openstack Operators] ********************************************** 2026-03-09 18:36:26.200646 | controller | 2026-03-09 18:36:26.200651 | controller | TASK [Use the locally built operators if any _local_operators_indexes={{ 2026-03-09 18:36:26.200656 | controller | _local_operators_indexes|default({}) | 2026-03-09 18:36:26.200660 | controller | combine({ item.key.split('-')[0]|upper+'_IMG': 2026-03-09 18:36:26.200665 | controller | cifmw_operator_build_output['operators'][item.key].image_catalog}) 2026-03-09 18:36:26.200669 | controller | }}] *** 2026-03-09 18:36:26.200674 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.059) 0:03:22.083 ********** 2026-03-09 18:36:26.200678 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.059) 0:03:22.082 ********** 2026-03-09 18:36:26.200688 | controller | ok: [controller] => (item={'key': 'openstack-operator', 'value': {'git_commit_hash': '581e9c0db82d502a54d54052822ffac7bccc0534', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/openstack-operator', 'image': '38.102.83.110:5001/openstack-k8s-operators/openstack-operator:581e9c0db82d502a54d54052822ffac7bccc0534', 'image_bundle': '38.102.83.110:5001/openstack-k8s-operators/openstack-operator-bundle:581e9c0db82d502a54d54052822ffac7bccc0534', 'image_catalog': '38.102.83.110:5001/openstack-k8s-operators/openstack-operator-index:581e9c0db82d502a54d54052822ffac7bccc0534'}}) 2026-03-09 18:36:26.227721 | controller | ok: [controller] => (item={'key': 'watcher-operator', 'value': {'git_commit_hash': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/watcher-operator', 'image': '38.102.83.110:5001/openstack-k8s-operators/watcher-operator:2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'image_bundle': '38.102.83.110:5001/openstack-k8s-operators/watcher-operator-bundle:2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'image_catalog': '38.102.83.110:5001/openstack-k8s-operators/watcher-operator-index:2e035aad6e396aeb72cc6aec8684c43e59f8b674'}}) 2026-03-09 18:36:26.227756 | controller | 2026-03-09 18:36:26.227763 | controller | TASK [Set install_yamls Makefile environment variables cifmw_edpm_prepare_common_env={{ 2026-03-09 18:36:26.227768 | controller | cifmw_install_yamls_environment | 2026-03-09 18:36:26.227773 | controller | combine({'PATH': cifmw_path}) | 2026-03-09 18:36:26.227777 | controller | combine(cifmw_edpm_prepare_extra_vars | default({})) 2026-03-09 18:36:26.227782 | controller | }}, cifmw_edpm_prepare_operators_build_output={{ operators_build_output }}, cifmw_edpm_prepare_make_openstack_env={{ _local_operators_indexes | combine(_openstack_operator_images) }}] *** 2026-03-09 18:36:26.227787 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.051) 0:03:22.135 ********** 2026-03-09 18:36:26.227792 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.051) 0:03:22.134 ********** 2026-03-09 18:36:26.227800 | controller | ok: [controller] 2026-03-09 18:36:26.576609 | controller | 2026-03-09 18:36:26.576643 | controller | TASK [detect if openstack operator is installed _raw_params=oc get sub --ignore-not-found=true -n openstack-operators -o name openstack-operator] *** 2026-03-09 18:36:26.576650 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.027) 0:03:22.162 ********** 2026-03-09 18:36:26.576655 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.027) 0:03:22.161 ********** 2026-03-09 18:36:26.576664 | controller | changed: [controller] 2026-03-09 18:36:26.605045 | controller | 2026-03-09 18:36:26.605082 | controller | TASK [Install openstack operator and wait for the csv to succeed name=install_yamls_makes, tasks_from=make_openstack_init] *** 2026-03-09 18:36:26.605091 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.348) 0:03:22.511 ********** 2026-03-09 18:36:26.605098 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.348) 0:03:22.510 ********** 2026-03-09 18:36:26.605109 | controller | 2026-03-09 18:36:26.644666 | controller | TASK [install_yamls_makes : Debug make_openstack_init_env var=make_openstack_init_env] *** 2026-03-09 18:36:26.644710 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.028) 0:03:22.539 ********** 2026-03-09 18:36:26.644719 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.028) 0:03:22.538 ********** 2026-03-09 18:36:26.644732 | controller | ok: [controller] => 2026-03-09 18:36:26.668799 | controller | make_openstack_init_env: 2026-03-09 18:36:26.668840 | controller | BMO_SETUP: false 2026-03-09 18:36:26.668850 | controller | CHECKOUT_FROM_OPENSTACK_REF: 'true' 2026-03-09 18:36:26.668857 | controller | KUBECONFIG: /home/zuul/.crc/machines/crc/kubeconfig 2026-03-09 18:36:26.668864 | controller | NETWORK_MTU: 1500 2026-03-09 18:36:26.668869 | controller | NNCP_DNS_SERVER: 192.168.122.10 2026-03-09 18:36:26.668875 | controller | NNCP_INTERFACE: ens7 2026-03-09 18:36:26.668882 | controller | OPENSTACK_BUNDLE_IMG: 38.102.83.110:5001/openstack-k8s-operators/openstack-operator-bundle:581e9c0db82d502a54d54052822ffac7bccc0534 2026-03-09 18:36:26.668891 | controller | OPENSTACK_IMG: 38.102.83.110:5001/openstack-k8s-operators/openstack-operator-index:581e9c0db82d502a54d54052822ffac7bccc0534 2026-03-09 18:36:26.668896 | controller | OPENSTACK_K8S_BRANCH: main 2026-03-09 18:36:26.668901 | controller | OUT: /home/zuul/ci-framework-data/artifacts/manifests 2026-03-09 18:36:26.668905 | controller | OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm 2026-03-09 18:36:26.668909 | controller | PATH: /home/zuul/.crc/bin:/home/zuul/.crc/bin/oc:/home/zuul/bin:/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin 2026-03-09 18:36:26.668914 | controller | WATCHER_BRANCH: '' 2026-03-09 18:36:26.668918 | controller | WATCHER_IMG: 38.102.83.110:5001/openstack-k8s-operators/watcher-operator-index:2e035aad6e396aeb72cc6aec8684c43e59f8b674 2026-03-09 18:36:26.668923 | controller | WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator 2026-03-09 18:36:26.668927 | controller | 2026-03-09 18:36:26.668932 | controller | TASK [install_yamls_makes : Debug make_openstack_init_params var=make_openstack_init_params] *** 2026-03-09 18:36:26.668937 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.039) 0:03:22.579 ********** 2026-03-09 18:36:26.668941 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.039) 0:03:22.577 ********** 2026-03-09 18:36:26.668950 | controller | skipping: [controller] 2026-03-09 18:41:14.098898 | controller | 2026-03-09 18:41:14.098939 | controller | TASK [install_yamls_makes : Run openstack_init output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls, script=make openstack_init, dry_run={{ make_openstack_init_dryrun|default(false)|bool }}, extra_args={{ dict((make_openstack_init_env|default({})), **(make_openstack_init_params|default({}))) }}] *** 2026-03-09 18:41:14.098949 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.024) 0:03:22.603 ********** 2026-03-09 18:41:14.098956 | controller | Monday 09 March 2026 18:36:26 +0000 (0:00:00.024) 0:03:22.602 ********** 2026-03-09 18:41:14.098968 | controller | [WARNING]: conditional statements should not include jinja2 templating 2026-03-09 18:41:14.119048 | controller | delimiters such as {{ }} or {% %}. Found: {{ make_openstack_init_until | 2026-03-09 18:41:14.119079 | controller | default(true) }} 2026-03-09 18:41:14.119092 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_003_run_openstack.log 2026-03-09 18:41:14.156832 | controller | changed: [controller] 2026-03-09 18:41:14.156873 | controller | 2026-03-09 18:41:14.156882 | controller | TASK [Run hooks after installing openstack name=run_hook] ********************** 2026-03-09 18:41:14.156890 | controller | Monday 09 March 2026 18:41:14 +0000 (0:04:47.450) 0:08:10.053 ********** 2026-03-09 18:41:14.156918 | controller | Monday 09 March 2026 18:41:14 +0000 (0:04:47.450) 0:08:10.052 ********** 2026-03-09 18:41:14.156932 | controller | 2026-03-09 18:41:14.239763 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-09 18:41:14.239804 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.037) 0:08:10.091 ********** 2026-03-09 18:41:14.239813 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.037) 0:08:10.089 ********** 2026-03-09 18:41:14.239825 | controller | ok: [controller] 2026-03-09 18:41:14.335430 | controller | 2026-03-09 18:41:14.335461 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-09 18:41:14.335468 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.083) 0:08:10.174 ********** 2026-03-09 18:41:14.335473 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.083) 0:08:10.173 ********** 2026-03-09 18:41:14.335482 | controller | ok: [controller] 2026-03-09 18:41:14.469860 | controller | 2026-03-09 18:41:14.469903 | controller | TASK [run_hook : Loop on hooks for post_install_operators_kuttl_from_operator _raw_params={{ hook.type }}.yml] *** 2026-03-09 18:41:14.469912 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.096) 0:08:10.270 ********** 2026-03-09 18:41:14.469919 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.095) 0:08:10.269 ********** 2026-03-09 18:41:14.469931 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Deploy watcher operator', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator/ci/playbooks/deploy_watcher_operator.yaml', 'extra_vars': {'content_provider_os_registry_url': '38.102.83.110:5001/podified-master-centos10', 'watcher_catalog_image': '38.102.83.110:5001/openstack-k8s-operators/watcher-operator-index:2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'watcher_services_tag': 'watcher_latest', 'watcher_repo': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}}) 2026-03-09 18:41:14.525977 | controller | 2026-03-09 18:41:14.526015 | controller | TASK [run_hook : Set playbook path for Deploy watcher operator cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] *** 2026-03-09 18:41:14.526026 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.134) 0:08:10.404 ********** 2026-03-09 18:41:14.526034 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.134) 0:08:10.403 ********** 2026-03-09 18:41:14.526046 | controller | ok: [controller] 2026-03-09 18:41:14.749773 | controller | 2026-03-09 18:41:14.749806 | controller | TASK [run_hook : Get file stat path={{ playbook_path }}] *********************** 2026-03-09 18:41:14.749838 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.056) 0:08:10.460 ********** 2026-03-09 18:41:14.749845 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.056) 0:08:10.459 ********** 2026-03-09 18:41:14.749856 | controller | ok: [controller] 2026-03-09 18:41:14.771805 | controller | 2026-03-09 18:41:14.771842 | controller | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] *** 2026-03-09 18:41:14.771851 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.223) 0:08:10.684 ********** 2026-03-09 18:41:14.771856 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.223) 0:08:10.683 ********** 2026-03-09 18:41:14.771865 | controller | skipping: [controller] 2026-03-09 18:41:14.981114 | controller | 2026-03-09 18:41:14.981149 | controller | TASK [run_hook : Get parameters files paths={{ 2026-03-09 18:41:14.981160 | controller | (cifmw_basedir, 'artifacts/parameters') | path_join 2026-03-09 18:41:14.981183 | controller | }}, file_type=file, patterns=*.yml] *** 2026-03-09 18:41:14.981191 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.021) 0:08:10.706 ********** 2026-03-09 18:41:14.981197 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.021) 0:08:10.705 ********** 2026-03-09 18:41:14.981206 | controller | ok: [controller] 2026-03-09 18:41:15.005623 | controller | 2026-03-09 18:41:15.005660 | controller | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] *** 2026-03-09 18:41:15.005667 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.209) 0:08:10.916 ********** 2026-03-09 18:41:15.005674 | controller | Monday 09 March 2026 18:41:14 +0000 (0:00:00.209) 0:08:10.914 ********** 2026-03-09 18:41:15.005687 | controller | ok: [controller] 2026-03-09 18:41:15.219894 | controller | 2026-03-09 18:41:15.219932 | controller | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] *** 2026-03-09 18:41:15.219940 | controller | Monday 09 March 2026 18:41:15 +0000 (0:00:00.024) 0:08:10.940 ********** 2026-03-09 18:41:15.219947 | controller | Monday 09 March 2026 18:41:15 +0000 (0:00:00.024) 0:08:10.939 ********** 2026-03-09 18:41:15.219959 | controller | ok: [controller] 2026-03-09 18:41:15.418344 | controller | 2026-03-09 18:41:15.418396 | controller | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-09 18:41:15.418410 | controller | Monday 09 March 2026 18:41:15 +0000 (0:00:00.214) 0:08:11.154 ********** 2026-03-09 18:41:15.418421 | controller | Monday 09 March 2026 18:41:15 +0000 (0:00:00.214) 0:08:11.153 ********** 2026-03-09 18:41:15.418439 | controller | ok: [controller] 2026-03-09 18:42:15.263551 | controller | 2026-03-09 18:42:15.263597 | controller | TASK [run_hook : Run hook without retry - Deploy watcher operator] ************* 2026-03-09 18:42:15.263607 | controller | Monday 09 March 2026 18:41:15 +0000 (0:00:00.198) 0:08:11.352 ********** 2026-03-09 18:42:15.263613 | controller | Monday 09 March 2026 18:41:15 +0000 (0:00:00.198) 0:08:11.351 ********** 2026-03-09 18:42:15.263625 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log 2026-03-09 18:42:15.286742 | controller | changed: [controller] 2026-03-09 18:42:15.286779 | controller | 2026-03-09 18:42:15.286786 | controller | TASK [run_hook : Run hook with retry - Deploy watcher operator] **************** 2026-03-09 18:42:15.286793 | controller | Monday 09 March 2026 18:42:15 +0000 (0:00:59.845) 0:09:11.198 ********** 2026-03-09 18:42:15.286799 | controller | Monday 09 March 2026 18:42:15 +0000 (0:00:59.845) 0:09:11.196 ********** 2026-03-09 18:42:15.286811 | controller | skipping: [controller] 2026-03-09 18:42:15.509609 | controller | 2026-03-09 18:42:15.509642 | controller | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-09 18:42:15.509649 | controller | Monday 09 March 2026 18:42:15 +0000 (0:00:00.023) 0:09:11.221 ********** 2026-03-09 18:42:15.509654 | controller | Monday 09 March 2026 18:42:15 +0000 (0:00:00.023) 0:09:11.220 ********** 2026-03-09 18:42:15.509663 | controller | ok: [controller] 2026-03-09 18:42:15.536361 | controller | 2026-03-09 18:42:15.536400 | controller | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-09 18:42:15.536408 | controller | Monday 09 March 2026 18:42:15 +0000 (0:00:00.222) 0:09:11.444 ********** 2026-03-09 18:42:15.536414 | controller | Monday 09 March 2026 18:42:15 +0000 (0:00:00.222) 0:09:11.443 ********** 2026-03-09 18:42:15.536426 | controller | skipping: [controller] 2026-03-09 18:44:55.412107 | controller | 2026-03-09 18:44:55.412169 | controller | TASK [install kuttl test_suite dependencies chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-prep] *** 2026-03-09 18:44:55.412179 | controller | Monday 09 March 2026 18:42:15 +0000 (0:00:00.026) 0:09:11.471 ********** 2026-03-09 18:44:55.412185 | controller | Monday 09 March 2026 18:42:15 +0000 (0:00:00.026) 0:09:11.469 ********** 2026-03-09 18:44:55.412206 | controller | changed: [controller] 2026-03-09 18:44:55.507660 | controller | 2026-03-09 18:44:55.507728 | controller | PLAY RECAP ********************************************************************* 2026-03-09 18:44:55.507735 | controller | controller : ok=116 changed=42 unreachable=0 failed=0 skipped=77 rescued=0 ignored=0 2026-03-09 18:44:55.507740 | controller | 2026-03-09 18:44:55.507744 | controller | Monday 09 March 2026 18:44:55 +0000 (0:02:39.875) 0:11:51.346 ********** 2026-03-09 18:44:55.507748 | controller | =============================================================================== 2026-03-09 18:44:55.507753 | controller | install_yamls_makes : Run openstack_init ------------------------------ 287.45s 2026-03-09 18:44:55.507757 | controller | install kuttl test_suite dependencies --------------------------------- 159.88s 2026-03-09 18:44:55.507762 | controller | run_hook : Run hook without retry - Deploy watcher operator ------------ 59.85s 2026-03-09 18:44:55.507766 | controller | openshift_obs : Wait for observability operator deployment ------------- 46.34s 2026-03-09 18:44:55.507770 | controller | install_yamls_makes : Run download_tools ------------------------------- 40.94s 2026-03-09 18:44:55.507775 | controller | run_hook : Run hook without retry - Download needed tools -------------- 35.78s 2026-03-09 18:44:55.507779 | controller | ci_setup : Install needed packages ------------------------------------- 30.69s 2026-03-09 18:44:55.507783 | controller | repo_setup : Initialize python venv and install requirements ------------ 7.93s 2026-03-09 18:44:55.507788 | controller | ci_setup : Install openshift client ------------------------------------- 5.70s 2026-03-09 18:44:55.507792 | controller | openshift_setup : Create required namespaces ---------------------------- 2.24s 2026-03-09 18:44:55.507797 | controller | ci_setup : Manage directories ------------------------------------------- 1.17s 2026-03-09 18:44:55.507801 | controller | Gathering Facts --------------------------------------------------------- 1.14s 2026-03-09 18:44:55.507811 | controller | repo_setup : Ensure directories are present ----------------------------- 1.10s 2026-03-09 18:44:55.507816 | controller | openshift_setup : Patch network operator -------------------------------- 1.08s 2026-03-09 18:44:55.507820 | controller | openshift_setup : Gather network.operator info -------------------------- 1.06s 2026-03-09 18:44:55.507824 | controller | repo_setup : Make sure git-core package is installed -------------------- 1.04s 2026-03-09 18:44:55.507828 | controller | openshift_obs : Wait for observability-operator pod --------------------- 0.98s 2026-03-09 18:44:55.507833 | controller | install_yamls : Create the install_yamls parameters file ---------------- 0.94s 2026-03-09 18:44:55.507837 | controller | openshift_setup : Patch samples registry configuration ------------------ 0.89s 2026-03-09 18:44:55.507842 | controller | repo_setup : Get repo-setup repository ---------------------------------- 0.87s 2026-03-09 18:44:55.507847 | controller | Monday 09 March 2026 18:44:55 +0000 (0:02:39.875) 0:11:51.345 ********** 2026-03-09 18:44:55.507854 | controller | =============================================================================== 2026-03-09 18:44:55.507860 | controller | install_yamls_makes --------------------------------------------------- 328.50s 2026-03-09 18:44:55.507866 | controller | ansible.builtin.command ----------------------------------------------- 160.22s 2026-03-09 18:44:55.507872 | controller | run_hook --------------------------------------------------------------- 98.98s 2026-03-09 18:44:55.507878 | controller | openshift_obs ---------------------------------------------------------- 48.17s 2026-03-09 18:44:55.507884 | controller | ci_setup --------------------------------------------------------------- 38.97s 2026-03-09 18:44:55.507890 | controller | repo_setup ------------------------------------------------------------- 18.10s 2026-03-09 18:44:55.507896 | controller | openshift_setup --------------------------------------------------------- 5.90s 2026-03-09 18:44:55.507902 | controller | openshift_login --------------------------------------------------------- 4.26s 2026-03-09 18:44:55.507909 | controller | install_yamls ----------------------------------------------------------- 3.93s 2026-03-09 18:44:55.507916 | controller | cifmw_setup ------------------------------------------------------------- 1.52s 2026-03-09 18:44:55.507930 | controller | gather_facts ------------------------------------------------------------ 1.14s 2026-03-09 18:44:55.507938 | controller | install_ca -------------------------------------------------------------- 0.49s 2026-03-09 18:44:55.507945 | controller | discover_latest_image --------------------------------------------------- 0.40s 2026-03-09 18:44:55.507951 | controller | networking_mapper ------------------------------------------------------- 0.28s 2026-03-09 18:44:55.507958 | controller | ansible.builtin.include_vars -------------------------------------------- 0.14s 2026-03-09 18:44:55.507965 | controller | ansible.builtin.set_fact ------------------------------------------------ 0.14s 2026-03-09 18:44:55.507972 | controller | ansible.builtin.include_role -------------------------------------------- 0.10s 2026-03-09 18:44:55.507978 | controller | ansible.builtin.assert -------------------------------------------------- 0.07s 2026-03-09 18:44:55.507984 | controller | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2026-03-09 18:44:55.507990 | controller | total ----------------------------------------------------------------- 711.31s 2026-03-09 18:44:55.673545 | controller | changed 2026-03-09 18:44:55.744176 | 2026-03-09 18:44:55.744292 | PLAY RECAP 2026-03-09 18:44:55.744348 | controller | ok: 1 changed: 1 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-09 18:44:55.744377 | 2026-03-09 18:44:55.845984 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-deps.yaml@main] 2026-03-09 18:44:55.850356 | RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-run.yaml@main] 2026-03-09 18:44:56.412665 | 2026-03-09 18:44:56.412769 | PLAY [Run ci/playbooks/kuttl/run-kuttl-from-operator-targets.yaml] 2026-03-09 18:44:56.437135 | 2026-03-09 18:44:56.437263 | TASK [Run kuttl tests from operator playbook] 2026-03-09 18:44:59.243791 | controller | 2026-03-09 18:44:59.278896 | controller | PLAY [controller] ************************************************************** 2026-03-09 18:44:59.278962 | controller | 2026-03-09 18:44:59.278968 | controller | TASK [Run hooks before running kuttl tests name=run_hook] ********************** 2026-03-09 18:44:59.278973 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.044) 0:00:00.045 ********** 2026-03-09 18:44:59.278978 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.044) 0:00:00.044 ********** 2026-03-09 18:44:59.278991 | controller | 2026-03-09 18:44:59.353529 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-09 18:44:59.353601 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.034) 0:00:00.079 ********** 2026-03-09 18:44:59.353608 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.034) 0:00:00.079 ********** 2026-03-09 18:44:59.353624 | controller | ok: [controller] 2026-03-09 18:44:59.501159 | controller | 2026-03-09 18:44:59.501221 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-09 18:44:59.501236 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.074) 0:00:00.154 ********** 2026-03-09 18:44:59.501242 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.074) 0:00:00.153 ********** 2026-03-09 18:44:59.501255 | controller | ok: [controller] 2026-03-09 18:44:59.656024 | controller | 2026-03-09 18:44:59.656084 | controller | TASK [run_hook : Loop on hooks for pre_kuttl_from_operator _raw_params={{ hook.type }}.yml] *** 2026-03-09 18:44:59.656091 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.147) 0:00:00.301 ********** 2026-03-09 18:44:59.656096 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.147) 0:00:00.301 ********** 2026-03-09 18:44:59.656109 | controller | skipping: [controller] 2026-03-09 19:10:03.508648 | controller | 2026-03-09 19:10:03.508728 | controller | TASK [run kuttl test suite from operator Makefile chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-run] *** 2026-03-09 19:10:03.508737 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.155) 0:00:00.457 ********** 2026-03-09 19:10:03.508744 | controller | Monday 09 March 2026 18:44:59 +0000 (0:00:00.155) 0:00:00.456 ********** 2026-03-09 19:10:03.508758 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/run-kuttl-from-operator-targets.yaml:16 2026-03-09 19:10:03.508804 | controller | fatal: [controller]: FAILED! => 2026-03-09 19:10:03.508816 | controller | changed: true 2026-03-09 19:10:03.508823 | controller | cmd: 2026-03-09 19:10:03.508829 | controller | - make 2026-03-09 19:10:03.508836 | controller | - kuttl-test-run 2026-03-09 19:10:03.508842 | controller | delta: '0:25:02.921189' 2026-03-09 19:10:03.508853 | controller | end: '2026-03-09 19:10:02.836855' 2026-03-09 19:10:03.508861 | controller | msg: non-zero return code 2026-03-09 19:10:03.508867 | controller | rc: 2 2026-03-09 19:10:03.508874 | controller | start: '2026-03-09 18:44:59.915666' 2026-03-09 19:10:03.508880 | controller | stderr: |- 2026-03-09 19:10:03.508886 | controller | Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`. 2026-03-09 19:10:03.508893 | controller | Warning: The 'cinderBackup' field is deprecated and will be removed in a future release. Please migrate to 'cinderBackups'. 2026-03-09 19:10:03.508899 | controller | make: *** [Makefile:444: kuttl-test-run] Error 1 2026-03-09 19:10:03.508905 | controller | stderr_lines: 2026-03-09 19:10:03.508912 | controller | - 'Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default 2026-03-09 19:10:03.508918 | controller | value changed from `Never` to `Always`.' 2026-03-09 19:10:03.508924 | controller | - 'Warning: The ''cinderBackup'' field is deprecated and will be removed in a future 2026-03-09 19:10:03.508930 | controller | release. Please migrate to ''cinderBackups''.' 2026-03-09 19:10:03.508936 | controller | - 'make: *** [Makefile:444: kuttl-test-run] Error 1' 2026-03-09 19:10:03.508955 | controller | stdout: "oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml\n=== 2026-03-09 19:10:03.508962 | controller | RUN kuttl\n harness.go:463: starting setup\n harness.go:255: running tests 2026-03-09 19:10:03.508968 | controller | using configured kubeconfig.\n harness.go:278: Successful connection to cluster 2026-03-09 19:10:03.508974 | controller | at: https://api.crc.testing:6443\n harness.go:363: running tests\n harness.go:75: 2026-03-09 19:10:03.508982 | controller | going to run test suite with timeout of 300 seconds for each step\n harness.go:375: 2026-03-09 19:10:03.508988 | controller | testsuite: test/kuttl/test-suites/default/ has 11 tests\n=== RUN kuttl/harness\n=== 2026-03-09 19:10:03.508994 | controller | RUN kuttl/harness/appcred-tests\n=== PAUSE kuttl/harness/appcred-tests\n=== RUN 2026-03-09 19:10:03.509000 | controller | \ kuttl/harness/common\n=== PAUSE kuttl/harness/common\n=== RUN kuttl/harness/deps\n=== 2026-03-09 19:10:03.509006 | controller | PAUSE kuttl/harness/deps\n=== RUN kuttl/harness/watcher\n=== PAUSE kuttl/harness/watcher\n=== 2026-03-09 19:10:03.509012 | controller | RUN kuttl/harness/watcher-api-scaling\n=== PAUSE kuttl/harness/watcher-api-scaling\n=== 2026-03-09 19:10:03.509018 | controller | RUN kuttl/harness/watcher-cinder\n=== PAUSE kuttl/harness/watcher-cinder\n=== 2026-03-09 19:10:03.509024 | controller | RUN kuttl/harness/watcher-notification\n=== PAUSE kuttl/harness/watcher-notification\n=== 2026-03-09 19:10:03.509030 | controller | RUN kuttl/harness/watcher-rmquser\n=== PAUSE kuttl/harness/watcher-rmquser\n=== 2026-03-09 19:10:03.509036 | controller | RUN kuttl/harness/watcher-tls\n=== PAUSE kuttl/harness/watcher-tls\n=== RUN kuttl/harness/watcher-tls-certs-change\n=== 2026-03-09 19:10:03.509042 | controller | PAUSE kuttl/harness/watcher-tls-certs-change\n=== RUN kuttl/harness/watcher-topology\n=== 2026-03-09 19:10:03.509048 | controller | PAUSE kuttl/harness/watcher-topology\n=== CONT kuttl/harness/appcred-tests\n logger.go:42: 2026-03-09 19:10:03.509054 | controller | 18:45:00 | appcred-tests | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-09 19:10:03.509060 | controller | \ logger.go:42: 18:45:00 | appcred-tests/0-cleanup | starting test step 0-cleanup\n 2026-03-09 19:10:03.509071 | controller | \ logger.go:42: 18:45:00 | appcred-tests/0-cleanup | test step completed 0-cleanup\n 2026-03-09 19:10:03.509077 | controller | \ logger.go:42: 18:45:00 | appcred-tests/1-deploy | starting test step 1-deploy\n 2026-03-09 19:10:03.509083 | controller | \ logger.go:42: 18:45:00 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.509089 | controller | created\n logger.go:42: 18:45:00 | appcred-tests/1-deploy | running command: 2026-03-09 19:10:03.509095 | controller | [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n oc wait -n \"${NS}\" 2026-03-09 19:10:03.509101 | controller | watcher watcher-kuttl --for=condition=Ready --timeout=600s]\n logger.go:42: 18:45:57 2026-03-09 19:10:03.509107 | controller | | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition 2026-03-09 19:10:03.509114 | controller | met\n logger.go:42: 18:45:57 | appcred-tests/1-deploy | test step completed 1-deploy\n 2026-03-09 19:10:03.509120 | controller | \ logger.go:42: 18:45:57 | appcred-tests/2-deploy-appcred | starting test step 2026-03-09 19:10:03.509126 | controller | 2-deploy-appcred\n logger.go:42: 18:45:57 | appcred-tests/2-deploy-appcred | 2026-03-09 19:10:03.509132 | controller | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.509138 | controller | \ old_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.509170 | controller | \ old_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.509181 | controller | \ old_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.509185 | controller | jsonpath='{.metadata.uid}')\n oc create configmap appcred-watcher-pre \\\n 2026-03-09 19:10:03.509199 | controller | \ --from-literal=api_uid=\"${old_api_uid}\" \\\n --from-literal=applier_uid=\"${old_applier_uid}\" 2026-03-09 19:10:03.509204 | controller | \\\n --from-literal=decision_engine_uid=\"${old_de_uid}\" \\\n --dry-run=client 2026-03-09 19:10:03.509208 | controller | -o yaml | oc apply -n \"${NS}\" -f -\n \n cat <<'EOF' | oc apply -n 2026-03-09 19:10:03.509212 | controller | \"${NS}\" -f -\n apiVersion: keystone.openstack.org/v1beta1\n kind: 2026-03-09 19:10:03.509217 | controller | KeystoneApplicationCredential\n metadata:\n name: ac-watcher\n spec:\n 2026-03-09 19:10:03.509221 | controller | \ secret: osp-secret\n passwordSelector: WatcherPassword\n userName: 2026-03-09 19:10:03.509226 | controller | watcher\n roles:\n - admin\n - service\n unrestricted: 2026-03-09 19:10:03.509230 | controller | false\n EOF\n \n oc patch watcher watcher-kuttl -n \"${NS}\" 2026-03-09 19:10:03.509234 | controller | --type=merge -p '{\"spec\":{\"auth\":{\"applicationCredentialSecret\":\"ac-watcher-secret\"}}}']\n 2026-03-09 19:10:03.509239 | controller | \ logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre 2026-03-09 19:10:03.509243 | controller | created\n logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher 2026-03-09 19:10:03.509248 | controller | created\n logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.509252 | controller | patched\n logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.509257 | controller | [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n echo \"Waiting 2026-03-09 19:10:03.509261 | controller | for KeystoneApplicationCredential ac-watcher to be Ready...\"\n oc wait -n 2026-03-09 19:10:03.509265 | controller | \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \n ac_id=$(oc 2026-03-09 19:10:03.509270 | controller | get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n if [ 2026-03-09 19:10:03.509276 | controller | -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.509281 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.509285 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.509290 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.509294 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.509299 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.509303 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.509307 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509312 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509316 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509323 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.509327 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.509332 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.509336 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.509341 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.509347 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.509352 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.509356 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.509361 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.509365 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.509370 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.509374 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.509378 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.509383 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.509390 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.509394 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.509399 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.509403 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.509407 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.509412 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.509416 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.509421 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.509425 | controller | expected application_credential_id\"]\n logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.509429 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.509434 | controller | 18:50:58 | appcred-tests/2-deploy-appcred | error: timed out waiting for the condition 2026-03-09 19:10:03.509438 | controller | on keystoneapplicationcredentials/ac-watcher\n logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.509442 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.509447 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.509451 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.509456 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.509460 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.509464 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.509469 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.509473 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.509477 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.509518 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.509526 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.509532 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509545 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509552 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509556 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.509561 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.509565 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.509570 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.509574 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.509579 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.509583 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.509588 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.509592 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.509597 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.509601 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.509606 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.509612 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.509618 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.509624 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.509630 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.509646 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.509653 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.509659 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.509664 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.509668 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.509673 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.509677 | controller | expected application_credential_id\"]\n logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.509681 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.509686 | controller | 18:50:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.509690 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.509694 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.509699 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.509703 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.509713 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.509720 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.509726 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.509732 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.509738 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.509744 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.509749 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.509755 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.509761 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509767 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509772 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.509776 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.509780 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.509785 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.509789 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.509793 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.509798 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.509802 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.509808 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.509814 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.509820 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.509827 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.509833 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.509839 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.509844 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.509851 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.509860 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.509866 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.509883 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.509888 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.509892 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.509899 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.509905 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.509915 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.509921 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.509927 | controller | 18:51:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.509933 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.509939 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.509945 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.509951 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.509957 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.509963 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.509969 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.509975 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.509981 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.509986 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.509992 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.509998 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.510005 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510010 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510016 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510022 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.510028 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.510034 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.510040 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.510048 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510054 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510060 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.510066 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.510072 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.510078 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.510084 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.510090 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.510096 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.510104 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.510114 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.510119 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.510125 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.510129 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.510142 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.510165 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.510169 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.510174 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.510178 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.510182 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.510187 | controller | 18:51:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.510191 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.510195 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.510200 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.510204 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.510208 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.510212 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.510217 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.510221 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.510225 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.510229 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.510234 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.510238 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.510245 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510250 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510255 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510259 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.510263 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.510268 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.510272 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.510276 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510283 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510288 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.510292 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.510296 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.510301 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.510305 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.510309 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.510314 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.510318 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.510322 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.510326 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.510331 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.510335 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.510339 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.510344 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.510356 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.510361 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.510365 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.510371 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.510375 | controller | 18:51:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.510380 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.510384 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.510388 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.510393 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.510400 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.510405 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.510409 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.510414 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.510418 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.510422 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.510427 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.510431 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.510435 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510443 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510447 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510451 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.510456 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.510460 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.510464 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.510469 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510473 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510477 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.510482 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.510486 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.510490 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.510495 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.510500 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.510505 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.510512 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.510517 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.510523 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.510530 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.510539 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.510545 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.510549 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.510554 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.510566 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.510571 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.510575 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.510580 | controller | 18:51:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.510584 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.510588 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.510593 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.510597 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.510606 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.510613 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.510619 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.510625 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.510631 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.510637 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.510643 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.510649 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.510654 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510658 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510663 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510667 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.510671 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.510676 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.510681 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.510686 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510690 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510695 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.510701 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.510707 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.510713 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.510719 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.510725 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.510731 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.510737 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.510743 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.510748 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.510755 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.510759 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.510764 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.510768 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.510772 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.510776 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.510784 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.510799 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.510806 | controller | 18:51:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.510812 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.510819 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.510825 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.510831 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.510837 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.510843 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.510849 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.510855 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.510861 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.510867 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.510872 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.510879 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.510884 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510889 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510895 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.510903 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.510910 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.510916 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.510922 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.510928 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510934 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.510940 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.510946 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.510952 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.510958 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.510963 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.510969 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.510975 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.510981 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.510992 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.510998 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.511004 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511010 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.511016 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511022 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.511028 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.511034 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.511040 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511046 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.511061 | controller | 18:51:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.511068 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511074 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.511080 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.511086 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.511092 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.511097 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.511102 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.511106 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.511110 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.511114 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.511119 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.511123 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.511127 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511132 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511136 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511140 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.511156 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.511163 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.511167 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.511172 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511179 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511183 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.511188 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.511192 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.511196 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.511201 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.511205 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.511209 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.511216 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.511220 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.511225 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.511229 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511233 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.511237 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511242 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.511246 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.511250 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.511255 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511259 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.511263 | controller | 18:51:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.511275 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511279 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.511284 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.511288 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.511292 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.511297 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.511301 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.511306 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.511310 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.511314 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.511319 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.511323 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.511327 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511334 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511338 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511343 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.511347 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.511351 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.511356 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.511360 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511364 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511369 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.511373 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.511377 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.511382 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.511386 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.511390 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.511394 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.511399 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.511403 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.511409 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.511414 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511418 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.511422 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.511431 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.511435 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.511440 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511444 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.511448 | controller | 18:51:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.511453 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511457 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.511468 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.511473 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.511480 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.511486 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.511490 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.511495 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.511499 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.511503 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.511507 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.511512 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.511516 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511520 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511525 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511529 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.511535 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.511539 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.511544 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.511548 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511552 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511560 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.511564 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.511568 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.511573 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.511577 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.511581 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.511586 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.511590 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.511596 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.511601 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.511605 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511609 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.511614 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511618 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.511622 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.511627 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.511646 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511650 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.511655 | controller | 18:51:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.511659 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511663 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.511668 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.511678 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.511683 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.511687 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.511692 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.511696 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.511700 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.511705 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.511709 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.511713 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.511718 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511722 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511726 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511731 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.511735 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.511739 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.511744 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.511748 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511752 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511757 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.511761 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.511765 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.511770 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.511774 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.511778 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.511784 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.511789 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.511795 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.511799 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.511804 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511808 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.511812 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.511817 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.511821 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.511825 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.511830 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511834 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.511838 | controller | 18:51:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.511843 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.511847 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.511851 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.511856 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.511868 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.511873 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.511878 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.511882 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.511887 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.511891 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.511896 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.511900 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.511904 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511924 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511931 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.511937 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.511943 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.511949 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.511955 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.511962 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511971 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.511977 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.511982 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.511988 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.511994 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.512000 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.512005 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.512011 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.512016 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.512022 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.512028 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.512035 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512041 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.512047 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512053 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.512060 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.512066 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.512071 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512078 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.512084 | controller | 18:51:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.512090 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512096 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.512102 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.512108 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.512113 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.512117 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.512131 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.512136 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.512140 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.512163 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.512171 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.512175 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.512180 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512186 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512191 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512195 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.512200 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.512204 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.512208 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.512213 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.512217 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.512222 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.512226 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.512230 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.512235 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.512239 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.512243 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.512248 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.512252 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.512256 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.512261 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.512265 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512270 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.512274 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512278 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.512283 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.512287 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.512291 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512296 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.512300 | controller | 18:51:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.512305 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512309 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.512313 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.512318 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.512324 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.512328 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.512332 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.512344 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.512349 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.512354 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.512358 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.512362 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.512367 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512371 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512376 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512381 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.512386 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.512390 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.512395 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.512399 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.512403 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.512408 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.512412 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.512417 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.512421 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.512427 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.512431 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.512436 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.512440 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.512444 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.512449 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.512453 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512457 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.512462 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512466 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.512471 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.512475 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.512481 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512485 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.512490 | controller | 18:51:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.512494 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512499 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.512503 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.512507 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.512512 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.512516 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.512520 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.512525 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.512536 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.512541 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.512546 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.512550 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.512555 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512559 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512564 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512568 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.512572 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.512577 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.512581 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.512586 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.512591 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.512596 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.512600 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.512605 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.512609 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.512613 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.512618 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.512622 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.512626 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.512632 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.512637 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.512642 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512646 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.512650 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512655 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.512660 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.512664 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.512669 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512673 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.512678 | controller | 18:51:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.512682 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512687 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.512691 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.512695 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.512700 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.512704 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.512708 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.512713 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.512718 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.512723 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.512734 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.512739 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.512744 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512748 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512752 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512757 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.512761 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.512766 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.512770 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.512776 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.512784 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.512791 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.512797 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.512803 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.512809 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.512815 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.512821 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.512825 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.512829 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.512834 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.512840 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.512845 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512849 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.512853 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.512858 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.512862 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.512866 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.512871 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512878 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.512884 | controller | 18:51:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.512890 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.512896 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.512902 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.512908 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.512914 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.512920 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.512924 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.512929 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.512933 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.512937 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.512942 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.512954 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.512959 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512967 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512974 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.512980 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.512986 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.512992 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.512998 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.513004 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513010 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513016 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.513022 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.513026 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.513033 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.513037 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.513041 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.513045 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.513050 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.513054 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.513059 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.513066 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.513072 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.513078 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.513084 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.513090 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.513096 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.513103 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.513110 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.513117 | controller | 18:51:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.513124 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.513131 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.513138 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.513143 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.513185 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.513192 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.513196 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.513201 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.513207 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.513213 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.513219 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.513226 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.513232 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513247 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513253 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513259 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.513265 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.513271 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.513277 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.513284 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513290 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513296 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.513302 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.513308 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.513314 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.513320 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.513326 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.513332 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.513338 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.513344 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.513350 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.513356 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.513362 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.513368 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.513374 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.513380 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.513386 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.513398 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.513404 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.513411 | controller | 18:51:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.513417 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.513423 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.513429 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.513437 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.513444 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.513450 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.513456 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.513462 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.513467 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.513474 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.513480 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.513485 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.513492 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513497 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513510 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513514 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.513519 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.513523 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.513528 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.513532 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513537 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513541 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.513545 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.513550 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.513554 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.513558 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.513563 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.513567 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.513571 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.513578 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.513583 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.513587 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.513591 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.513596 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.513600 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.513605 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.513609 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.513613 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.513619 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.513623 | controller | 18:51:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.513628 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.513632 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.513636 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.513641 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.513645 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.513649 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.513654 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.513658 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.513662 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.513667 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.513671 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.513675 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.513680 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513684 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513688 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513693 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.513705 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.513713 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.513719 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.513725 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513736 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513742 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.513747 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.513754 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.513760 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.513766 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.513772 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.513777 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.513783 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.513789 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.513795 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.513801 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.513807 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.513813 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.513819 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.513825 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.513831 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.513836 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.513840 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.513844 | controller | 18:51:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.513849 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.513853 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.513858 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.513862 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.513866 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.513871 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.513876 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.513882 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.513889 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.513895 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.513901 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.513907 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.513913 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513923 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513929 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.513937 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.513941 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.513955 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.513960 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.513964 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513969 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.513974 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.513981 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.513987 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.513994 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.514000 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.514006 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.514012 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.514018 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.514024 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.514030 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.514036 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.514042 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.514048 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.514054 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.514058 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.514063 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.514067 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.514075 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.514082 | controller | 18:51:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.514088 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.514097 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.514105 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.514111 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.514120 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.514183 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.514194 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.514200 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.514206 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.514212 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.514218 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.514224 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.514230 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514236 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514242 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514248 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.514254 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.514260 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.514276 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.514283 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.514289 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.514294 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.514300 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.514306 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.514312 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.514318 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.514324 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.514330 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.514336 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.514342 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.514348 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.514354 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.514360 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.514366 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.514372 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.514378 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.514384 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.514394 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.514400 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.514406 | controller | 18:51:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.514412 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.514418 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.514424 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.514430 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.514436 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.514442 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.514448 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.514454 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.514460 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.514466 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.514472 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.514478 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.514484 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514491 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514497 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514502 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.514509 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.514514 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.514520 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.514538 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.514545 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.514551 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.514557 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.514563 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.514569 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.514574 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.514580 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.514586 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.514592 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.514603 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.514611 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.514617 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.514623 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.514629 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.514635 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.514641 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.514646 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.514652 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.514658 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.514664 | controller | 18:51:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.514670 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.514676 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.514682 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.514688 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.514694 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.514700 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.514706 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.514712 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.514718 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.514723 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.514729 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.514735 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.514741 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514747 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514753 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.514759 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.514765 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.514771 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.514777 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.514783 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.514791 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.514808 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.514814 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.514820 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.514826 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.514832 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.514838 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.514844 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.514850 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.514855 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.514861 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.514867 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.514873 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.514879 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.514885 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.514891 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.514897 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.514902 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.514908 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.514914 | controller | 18:51:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.514920 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.514928 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.514934 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.514940 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.514946 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.514952 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.514957 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.514963 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.514969 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.514975 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.514981 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.514987 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.514993 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515002 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515008 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515014 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.515020 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.515025 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.515031 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.515037 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515045 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515051 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.515067 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.515073 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.515079 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.515085 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.515091 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.515097 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.515102 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.515108 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.515114 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.515120 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515126 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.515132 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515138 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.515142 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.515161 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.515165 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515170 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.515174 | controller | 18:51:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.515178 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515182 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.515187 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.515191 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.515200 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.515205 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.515209 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.515213 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.515218 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.515222 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.515226 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.515230 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.515235 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515239 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515243 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515247 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.515253 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.515257 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.515262 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.515266 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515270 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515275 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.515279 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.515283 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.515295 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.515299 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.515304 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.515308 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.515312 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.515317 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.515321 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.515325 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515330 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.515334 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515338 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.515343 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.515347 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.515353 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515359 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.515366 | controller | 18:51:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.515372 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515378 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.515384 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.515390 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.515394 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.515400 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.515406 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.515412 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.515418 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.515424 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.515430 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.515435 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.515441 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515447 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515453 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515458 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.515462 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.515466 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.515470 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.515475 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515479 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515483 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.515487 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.515491 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.515496 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.515509 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.515513 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.515518 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.515522 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.515529 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.515533 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.515538 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515542 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.515546 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515550 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.515555 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.515559 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.515563 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515567 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.515572 | controller | 18:51:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.515578 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515582 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.515587 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.515591 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.515595 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.515599 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.515604 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.515608 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.515612 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.515616 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.515621 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.515625 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.515629 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515633 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515638 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515643 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.515647 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.515651 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.515656 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.515660 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515666 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515670 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.515675 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.515679 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.515683 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.515688 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.515692 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.515703 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.515708 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.515712 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.515717 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.515721 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515725 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.515730 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515734 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.515739 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.515743 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.515747 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515751 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.515755 | controller | 18:51:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.515760 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515764 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.515768 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.515772 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.515777 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.515781 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.515785 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.515790 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.515794 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.515798 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.515803 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.515807 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.515811 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515817 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515823 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.515828 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.515832 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.515836 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.515841 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.515845 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515849 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.515854 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.515858 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.515862 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.515867 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.515871 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.515875 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.515879 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.515891 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.515895 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.515900 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.515917 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515922 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.515926 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.515931 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.515935 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.515939 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.515944 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515948 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.515952 | controller | 18:51:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.515957 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.515961 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.515965 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.515970 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.515977 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.515982 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.515986 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.515991 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.515995 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.515999 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.516003 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.516008 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.516012 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516016 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516021 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516025 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.516029 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.516033 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.516038 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.516042 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516048 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516053 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.516057 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.516061 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.516066 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.516070 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.516074 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.516078 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.516083 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.516095 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.516100 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.516104 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.516108 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.516113 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.516117 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.516121 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.516126 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.516132 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.516137 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.516142 | controller | 18:51:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.516167 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.516174 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.516180 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.516186 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.516192 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.516196 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.516201 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.516205 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.516209 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.516214 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.516218 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.516222 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.516227 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516231 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516235 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516239 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.516244 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.516250 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.516256 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.516263 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516269 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516275 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.516281 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.516287 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.516293 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.516298 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.516302 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.516307 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.516311 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.516318 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.516323 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.516337 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.516342 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.516348 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.516355 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.516361 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.516367 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.516373 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.516379 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.516385 | controller | 18:51:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.516391 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.516397 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.516403 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.516409 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.516414 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.516420 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.516426 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.516431 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.516435 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.516441 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.516447 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.516453 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.516459 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516467 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516473 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516479 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.516485 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.516491 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.516497 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.516503 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516512 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516518 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.516524 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.516530 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.516536 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.516542 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.516548 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.516554 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.516560 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.516566 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.516572 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.516578 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.516595 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.516601 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.516607 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.516613 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.516619 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.516625 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.516631 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.516637 | controller | 18:51:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.516643 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.516649 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.516655 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.516661 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.516669 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.516676 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.516682 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.516688 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.516694 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.516699 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.516705 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.516711 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.516717 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516726 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516733 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516739 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.516745 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.516751 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.516757 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.516762 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516768 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516774 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.516780 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.516786 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.516792 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.516798 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.516804 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.516810 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.516815 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.516821 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.516827 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.516833 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.516837 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.516850 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.516854 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.516860 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.516864 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.516869 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.516873 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.516878 | controller | 18:51:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.516882 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.516886 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.516890 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.516895 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.516901 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.516906 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.516910 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.516914 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.516918 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.516922 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.516927 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.516931 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.516935 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516940 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516944 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.516948 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.516952 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.516956 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.516961 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.516965 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516969 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.516973 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.516978 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.516982 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.516986 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.516990 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.516994 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.516999 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.517003 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.517007 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.517011 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.517016 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517020 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.517024 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517028 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.517039 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.517044 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.517050 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517054 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.517059 | controller | 18:51:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.517063 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517067 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.517072 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.517076 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.517080 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.517084 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.517089 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.517093 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.517097 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.517101 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.517106 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.517110 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.517115 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517119 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517124 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517129 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.517133 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.517137 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.517142 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.517166 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.517173 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.517177 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.517181 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.517185 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.517190 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.517194 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.517198 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.517202 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.517207 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.517214 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.517218 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.517223 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517227 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.517232 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517236 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.517240 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.517252 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.517256 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517261 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.517265 | controller | 18:51:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.517269 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517274 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.517278 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.517282 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.517286 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.517291 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.517295 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.517299 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.517303 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.517308 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.517312 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.517316 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.517320 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517324 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517329 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517333 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.517339 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.517345 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.517351 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.517357 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.517365 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.517372 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.517377 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.517382 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.517386 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.517390 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.517395 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.517399 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.517403 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.517407 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.517412 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.517416 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517422 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.517426 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517431 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.517437 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.517443 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.517458 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517465 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.517471 | controller | 18:51:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.517477 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517483 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.517487 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.517492 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.517496 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.517500 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.517504 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.517509 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.517513 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.517517 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.517521 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.517526 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.517531 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517544 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517551 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517557 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.517563 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.517569 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.517575 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.517580 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.517586 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.517591 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.517595 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.517599 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.517604 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.517608 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.517612 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.517616 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.517621 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.517627 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.517635 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.517641 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517648 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.517654 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517660 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.517666 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.517671 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.517677 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517683 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.517699 | controller | 18:51:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.517705 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517710 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.517714 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.517719 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.517729 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.517735 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.517741 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.517747 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.517753 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.517759 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.517765 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.517773 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.517779 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517785 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517791 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.517797 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.517803 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.517808 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.517814 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.517821 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.517827 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.517833 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.517839 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.517845 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.517851 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.517857 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.517863 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.517868 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.517874 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.517880 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.517886 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.517892 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517898 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.517904 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.517910 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.517916 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.517922 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.517931 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517937 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.517943 | controller | 18:51:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.517982 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.517989 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.517996 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.518002 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.518008 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.518014 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.518020 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.518026 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.518032 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.518038 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.518044 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.518050 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.518056 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518061 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518067 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518073 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.518079 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.518085 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.518091 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.518097 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518102 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518108 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.518114 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.518120 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.518126 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.518132 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.518138 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.518171 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.518183 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.518193 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.518199 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.518205 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.518211 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.518217 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.518223 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.518229 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.518235 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.518241 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.518247 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.518253 | controller | 18:51:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.518258 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.518275 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.518281 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.518288 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.518308 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.518316 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.518322 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.518328 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.518334 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.518340 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.518345 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.518351 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.518357 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518363 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518369 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518375 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.518381 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.518387 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.518392 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.518398 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518409 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518414 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.518418 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.518422 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.518426 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.518431 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.518435 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.518439 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.518444 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.518448 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.518452 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.518456 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.518461 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.518465 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.518469 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.518473 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.518478 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.518482 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.518486 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.518490 | controller | 18:51:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.518495 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.518499 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.518503 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.518516 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.518520 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.518524 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.518529 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.518533 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.518537 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.518542 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.518548 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.518552 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.518556 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518562 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518567 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518571 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.518575 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.518580 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.518584 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.518588 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518592 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518597 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.518601 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.518605 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.518609 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.518614 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.518618 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.518622 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.518626 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.518630 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.518635 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.518639 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.518643 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.518647 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.518652 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.518656 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.518660 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.518664 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.518669 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.518673 | controller | 18:51:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.518677 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.518682 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.518686 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.518690 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.518702 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.518707 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.518711 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.518716 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.518720 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.518724 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.518728 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.518733 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.518737 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518741 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518745 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518751 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.518756 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.518760 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.518765 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.518769 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518773 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518777 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.518781 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.518786 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.518790 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.518794 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.518798 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.518804 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.518808 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.518813 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.518817 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.518821 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.518825 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.518829 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.518834 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.518838 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.518842 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.518849 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.518853 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.518857 | controller | 18:51:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.518862 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.518866 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.518870 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.518875 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.518879 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.518883 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.518894 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.518899 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.518903 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.518907 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.518911 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.518916 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.518920 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518924 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518929 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.518933 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.518937 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.518941 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.518946 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.518950 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518954 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.518958 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.518963 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.518967 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.518971 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.518976 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.518980 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.518984 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.518988 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.518994 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.518999 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.519005 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519011 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.519017 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519023 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.519029 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.519035 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.519040 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.519045 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.519049 | controller | 18:51:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.519053 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.519058 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.519062 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.519066 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.519070 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.519075 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.519079 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.519091 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.519098 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.519107 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.519114 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.519120 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.519125 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519131 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519137 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519143 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.519169 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.519176 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.519182 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.519187 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.519196 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.519202 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.519208 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.519214 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.519219 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.519225 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.519231 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.519237 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.519242 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.519248 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.519257 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.519263 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519269 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.519275 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519281 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.519287 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.519293 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.519298 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.519306 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.519312 | controller | 18:51:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.519319 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.519325 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.519330 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.519336 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.519340 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.519344 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.519349 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.519353 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.519368 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.519372 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.519377 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.519381 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.519386 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519392 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519397 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519401 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.519405 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.519410 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.519414 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.519418 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.519425 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.519429 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.519434 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.519438 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.519442 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.519450 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.519454 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.519458 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.519463 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.519467 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.519471 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.519475 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519480 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.519484 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519489 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.519495 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.519501 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.519507 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.519513 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.519519 | controller | 18:51:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.519525 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.519531 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.519537 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.519543 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.519552 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.519558 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.519564 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.519570 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.519576 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.519582 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.519598 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.519605 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.519611 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519617 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519623 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519629 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.519635 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.519639 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.519644 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.519648 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.519653 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.519659 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.519665 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.519671 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.519677 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.519683 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.519689 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.519695 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.519701 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.519707 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.519713 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.519719 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519725 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.519731 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519737 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.519743 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.519748 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.519758 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.519762 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.519766 | controller | 18:51:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.519772 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.519778 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.519785 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.519791 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.519800 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.519807 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.519813 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.519819 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.519827 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.519833 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.519839 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.519855 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.519861 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519868 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519874 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.519880 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.519886 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.519892 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.519898 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.519916 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.519923 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.519928 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.519934 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.519940 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.519946 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.519952 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.519958 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.519964 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.519970 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.519979 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.519986 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.519991 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.519997 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.520003 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520009 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.520015 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.520021 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.520026 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.520032 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.520038 | controller | 18:51:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.520044 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.520050 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.520056 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.520062 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.520068 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.520077 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.520083 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.520089 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.520095 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.520101 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.520107 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.520113 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.520119 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520135 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520142 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520167 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.520174 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.520180 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.520186 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.520192 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.520200 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.520206 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.520212 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.520218 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.520227 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.520233 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.520239 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.520245 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.520251 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.520257 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.520263 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.520269 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520275 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.520280 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520286 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.520292 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.520298 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.520304 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.520310 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.520316 | controller | 18:51:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.520322 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.520328 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.520334 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.520340 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.520346 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.520351 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.520357 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.520363 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.520369 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.520375 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.520381 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.520387 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.520392 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520406 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520423 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520434 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.520440 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.520446 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.520452 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.520458 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.520464 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.520469 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.520475 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.520481 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.520487 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.520493 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.520499 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.520505 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.520511 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.520516 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.520522 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.520528 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520534 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.520540 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520546 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.520552 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.520558 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.520564 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.520570 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.520576 | controller | 18:51:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.520582 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.520588 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.520593 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.520602 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.520611 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.520617 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.520623 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.520629 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.520635 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.520641 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.520647 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.520653 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.520659 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520665 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520671 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520677 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.520692 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.520698 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.520704 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.520710 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.520719 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.520725 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.520731 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.520737 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.520743 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.520749 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.520755 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.520761 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.520766 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.520772 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.520778 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.520784 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520790 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.520796 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520802 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.520808 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.520813 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.520822 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.520826 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.520830 | controller | 18:51:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.520835 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.520839 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.520843 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.520848 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.520852 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.520856 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.520861 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.520865 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.520869 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.520873 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.520878 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.520882 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.520886 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520890 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520895 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.520899 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.520903 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.520914 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.520919 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.520923 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.520928 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.520932 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.520936 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.520944 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.520949 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.520953 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.520957 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.520962 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.520966 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.520972 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.520978 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.520983 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520987 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.520991 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.520996 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.521000 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.521004 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.521008 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.521013 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.521017 | controller | 18:51:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.521022 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.521026 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.521032 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.521038 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.521044 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.521050 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.521056 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.521062 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.521068 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.521074 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.521080 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.521086 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.521092 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521098 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521104 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521110 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.521116 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.521122 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.521138 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.521158 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.521180 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.521185 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.521190 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.521194 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.521198 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.521202 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.521207 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.521213 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.521219 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.521225 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.521231 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.521237 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.521243 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.521248 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.521254 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.521260 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.521266 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.521271 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.521277 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.521283 | controller | 18:51:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.521289 | controller | \"ac-watcher\" not found\n logger.go:42: 18:51:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.521295 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.521303 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.521308 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.521313 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.521319 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.521325 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.521331 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.521337 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.521343 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.521349 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.521355 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.521361 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521370 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521375 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521381 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.521387 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.521393 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.521397 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.521413 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.521420 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.521426 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.521432 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.521438 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.521444 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.521450 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.521456 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.521461 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.521467 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.521473 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.521479 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.521485 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.521491 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.521497 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.521503 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.521509 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.521515 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.521521 | controller | expected application_credential_id\"]\n logger.go:42: 18:51:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.521527 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.521533 | controller | 18:52:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.521539 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.521545 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.521550 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.521556 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.521568 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.521574 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.521580 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.521586 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.521592 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.521598 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.521604 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.521610 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.521616 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521622 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521628 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521633 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.521639 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.521645 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.521651 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.521657 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.521675 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.521682 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.521689 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.521695 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.521701 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.521706 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.521712 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.521718 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.521724 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.521730 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.521736 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.521742 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.521748 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.521754 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.521759 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.521765 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.521771 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.521781 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.521787 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.521792 | controller | 18:52:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.521798 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.521804 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.521810 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.521816 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.521822 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.521828 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.521833 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.521839 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.521845 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.521850 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.521856 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.521862 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.521868 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521874 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521880 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.521886 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.521892 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.521898 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.521904 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.521910 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.521919 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.521926 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.521943 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.521951 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.521957 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.521963 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.521969 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.521974 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.521980 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.521990 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.521996 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.522002 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522008 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.522014 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522020 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.522025 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.522031 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.522037 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522043 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.522049 | controller | 18:52:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.522055 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522064 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.522070 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.522076 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.522082 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.522090 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.522096 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.522102 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.522109 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.522115 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.522121 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.522127 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.522133 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522139 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522167 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522181 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.522191 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.522197 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.522204 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.522210 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.522221 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.522228 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.522234 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.522240 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.522257 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.522263 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.522269 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.522275 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.522281 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.522287 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.522293 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.522298 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522304 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.522310 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522316 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.522322 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.522328 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.522333 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522338 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.522342 | controller | 18:52:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.522346 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522351 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.522355 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.522359 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.522363 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.522368 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.522372 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.522376 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.522381 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.522385 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.522389 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.522393 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.522398 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522404 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522409 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522413 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.522419 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.522424 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.522428 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.522432 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.522437 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.522441 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.522445 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.522450 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.522455 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.522470 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.522478 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.522484 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.522490 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.522494 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.522498 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.522503 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522507 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.522511 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522516 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.522520 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.522524 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.522529 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522533 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.522537 | controller | 18:52:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.522542 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522546 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.522550 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.522555 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.522561 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.522566 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.522570 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.522574 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.522578 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.522583 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.522587 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.522591 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.522596 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522600 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522604 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522609 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.522613 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.522617 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.522622 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.522626 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.522630 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.522634 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.522639 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.522643 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.522647 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.522652 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.522662 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.522667 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.522671 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.522676 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.522682 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.522687 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522691 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.522696 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522700 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.522704 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.522708 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.522715 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522720 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.522724 | controller | 18:52:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.522731 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522735 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.522740 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.522746 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.522752 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.522758 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.522764 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.522770 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.522776 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.522782 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.522788 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.522794 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.522798 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522802 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522807 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.522811 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.522815 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.522820 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.522824 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.522828 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.522833 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.522837 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.522841 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.522845 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.522850 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.522854 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.522858 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.522862 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.522875 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.522883 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.522889 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.522895 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522901 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.522907 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.522913 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.522919 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.522925 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.522931 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522937 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.522941 | controller | 18:52:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.522946 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.522950 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.522954 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.522959 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.522963 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.522967 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.522972 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.522976 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.522980 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.522986 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.522993 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.522999 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.523005 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523011 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523020 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523026 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.523032 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.523039 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.523044 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.523051 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.523063 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.523070 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.523076 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.523082 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.523088 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.523094 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.523100 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.523107 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.523113 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.523130 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.523137 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.523143 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.523165 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.523170 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.523174 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.523179 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.523183 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.523188 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.523195 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.523201 | controller | 18:52:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.523207 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.523213 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.523219 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.523226 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.523235 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.523241 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.523247 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.523253 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.523259 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.523265 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.523271 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.523275 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.523279 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523287 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523294 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523300 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.523306 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.523312 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.523318 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.523324 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.523330 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.523336 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.523342 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.523348 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.523354 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.523359 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.523365 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.523371 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.523377 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.523386 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.523393 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.523410 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.523416 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.523422 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.523428 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.523434 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.523440 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.523446 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.523454 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.523460 | controller | 18:52:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.523466 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.523472 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.523478 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.523484 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.523495 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.523501 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.523507 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.523513 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.523519 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.523525 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.523531 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.523536 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.523542 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523548 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523554 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523560 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.523566 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.523571 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.523577 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.523581 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.523586 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.523590 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.523594 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.523598 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.523603 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.523607 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.523611 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.523616 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.523620 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.523624 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.523628 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.523633 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.523648 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.523655 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.523662 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.523668 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.523674 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.523683 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.523689 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.523695 | controller | 18:52:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.523701 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.523706 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.523712 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.523718 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.523724 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.523730 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.523736 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.523742 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.523748 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.523754 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.523760 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.523766 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.523772 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523780 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523787 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.523795 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.523801 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.523807 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.523813 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.523818 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.523824 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.523829 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.523835 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.523841 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.523846 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.523852 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.523858 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.523864 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.523870 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.523878 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.523884 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.523890 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.523913 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.523930 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.523937 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.523943 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.523949 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.523955 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.523961 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.523967 | controller | 18:52:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.523973 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.523978 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.523984 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.523990 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.523996 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.524001 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.524007 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.524012 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.524018 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.524024 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.524030 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.524036 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.524042 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524048 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524054 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524060 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.524066 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.524072 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.524078 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.524084 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524093 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524099 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.524105 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.524110 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.524116 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.524122 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.524128 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.524133 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.524139 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.524159 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.524169 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.524175 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.524181 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.524187 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.524207 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.524213 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.524220 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.524225 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.524231 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.524237 | controller | 18:52:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.524244 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.524250 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.524256 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.524262 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.524267 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.524273 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.524279 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.524285 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.524291 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.524297 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.524303 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.524309 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.524314 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524323 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524330 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524336 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.524342 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.524348 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.524354 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.524360 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524368 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524374 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.524380 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.524386 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.524392 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.524398 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.524404 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.524410 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.524416 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.524422 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.524429 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.524435 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.524441 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.524447 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.524453 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.524459 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.524473 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.524480 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.524486 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.524492 | controller | 18:52:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.524498 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.524504 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.524509 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.524515 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.524523 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.524528 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.524532 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.524536 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.524540 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.524545 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.524549 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.524553 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.524560 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524564 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524568 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524573 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.524577 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.524581 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.524586 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.524590 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524594 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524598 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.524603 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.524607 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.524611 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.524616 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.524620 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.524624 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.524628 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.524633 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.524637 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.524641 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.524646 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.524650 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.524654 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.524659 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.524663 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.524676 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.524681 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.524685 | controller | 18:52:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.524690 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.524694 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.524698 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.524703 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.524707 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.524711 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.524716 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.524720 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.524724 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.524729 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.524733 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.524737 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.524741 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524746 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524750 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524754 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.524759 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.524763 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.524767 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.524772 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524776 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524780 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.524785 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.524789 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.524793 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.524797 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.524802 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.524806 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.524810 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.524816 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.524821 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.524825 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.524831 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.524835 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.524840 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.524844 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.524848 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.524852 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.524860 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.524871 | controller | 18:52:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.524875 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.524880 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.524884 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.524888 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.524894 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.524899 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.524903 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.524907 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.524912 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.524916 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.524920 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.524924 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.524929 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524933 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524937 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.524941 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.524946 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.524950 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.524954 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.524959 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524965 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.524969 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.524973 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.524978 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.524982 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.524986 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.524990 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.524995 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.524999 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.525003 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.525008 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.525012 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525016 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.525020 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525025 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.525029 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.525033 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.525038 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525042 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.525046 | controller | 18:52:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.525057 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525061 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.525065 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.525070 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.525074 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.525078 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.525083 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.525087 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.525091 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.525095 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.525100 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.525106 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.525110 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525117 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525121 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525126 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.525130 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.525134 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.525139 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.525143 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525165 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525172 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.525178 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.525184 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.525190 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.525195 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.525200 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.525204 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.525208 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.525212 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.525217 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.525221 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525225 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.525233 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525238 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.525242 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.525246 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.525250 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525255 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.525259 | controller | 18:52:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.525264 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525276 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.525281 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.525285 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.525292 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.525296 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.525301 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.525305 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.525309 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.525314 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.525318 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.525322 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.525326 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525331 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525335 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525341 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.525346 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.525350 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.525354 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.525358 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525364 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525369 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.525373 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.525377 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.525382 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.525386 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.525390 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.525394 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.525400 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.525404 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.525409 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.525413 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525417 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.525422 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525426 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.525430 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.525435 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.525440 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525445 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.525449 | controller | 18:52:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.525453 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525458 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.525462 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.525474 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.525479 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.525486 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.525492 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.525498 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.525504 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.525509 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.525513 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.525518 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.525522 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525526 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525531 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525535 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.525539 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.525544 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.525548 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.525552 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525557 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525561 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.525565 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.525569 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.525574 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.525578 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.525582 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.525587 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.525591 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.525597 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.525602 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.525606 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525610 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.525615 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525619 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.525623 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.525628 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.525632 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525636 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.525640 | controller | 18:52:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.525645 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525649 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.525653 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.525658 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.525669 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.525674 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.525678 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.525683 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.525687 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.525691 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.525698 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.525702 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.525706 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525711 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525715 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525719 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.525724 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.525728 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.525732 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.525737 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525743 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525747 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.525751 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.525756 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.525760 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.525764 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.525769 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.525773 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.525778 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.525784 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.525792 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.525798 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525805 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.525811 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.525817 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.525823 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.525828 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.525832 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525837 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.525841 | controller | 18:52:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.525845 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.525850 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.525854 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.525858 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.525863 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.525875 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.525881 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.525887 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.525894 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.525900 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.525906 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.525912 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.525918 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525929 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525934 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.525938 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.525943 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.525947 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.525951 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.525956 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525960 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.525964 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.525969 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.525973 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.525978 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.525985 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.525991 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.526000 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.526007 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.526013 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.526019 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.526025 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526031 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.526036 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526041 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.526045 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.526049 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.526054 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526058 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.526062 | controller | 18:52:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.526067 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526072 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.526078 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.526084 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.526094 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.526100 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.526106 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.526122 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.526128 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.526134 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.526140 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.526157 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.526165 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526171 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526177 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526183 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.526189 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.526195 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.526201 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.526207 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.526213 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.526219 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.526225 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.526231 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.526236 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.526242 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.526248 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.526254 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.526260 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.526266 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.526272 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.526278 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526284 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.526290 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526295 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.526301 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.526307 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.526317 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526322 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.526328 | controller | 18:52:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.526334 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526340 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.526346 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.526352 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.526361 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.526367 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.526371 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.526376 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.526388 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.526395 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.526399 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.526404 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.526408 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526412 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526416 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526420 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.526425 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.526429 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.526433 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.526437 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.526443 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.526447 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.526452 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.526456 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.526460 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.526464 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.526469 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.526473 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.526477 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.526484 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.526489 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.526493 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526497 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.526501 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526506 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.526510 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.526514 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.526518 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526523 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.526527 | controller | 18:52:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.526531 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526535 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.526540 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.526544 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.526548 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.526552 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.526556 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.526561 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.526565 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.526569 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.526580 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.526584 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.526588 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526593 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526597 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526601 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.526606 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.526610 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.526614 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.526618 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.526624 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.526629 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.526633 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.526637 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.526641 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.526647 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.526651 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.526656 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.526660 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.526664 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.526669 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.526673 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526677 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.526681 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526686 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.526690 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.526694 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.526698 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526703 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.526707 | controller | 18:52:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.526711 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526716 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.526721 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.526727 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.526733 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.526739 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.526745 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.526751 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.526757 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.526762 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.526767 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.526781 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.526786 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526792 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526797 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.526803 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.526808 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.526813 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.526819 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.526825 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.526831 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.526837 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.526843 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.526849 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.526855 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.526861 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.526865 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.526870 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.526874 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.526878 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.526882 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.526887 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526891 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.526895 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.526899 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.526904 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.526909 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.526914 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526921 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.526926 | controller | 18:52:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.526932 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.526938 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.526944 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.526950 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.526959 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.526965 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.526969 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.526973 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.526980 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.526984 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.526988 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.526993 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.526997 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527013 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527020 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527026 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.527031 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.527037 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.527043 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.527049 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.527054 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.527060 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.527066 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.527072 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.527078 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.527084 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.527089 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.527095 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.527101 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.527106 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.527110 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.527114 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.527120 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.527126 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.527132 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.527138 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.527158 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.527176 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.527182 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.527188 | controller | 18:52:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.527194 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.527200 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.527205 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.527211 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.527217 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.527226 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.527232 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.527238 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.527244 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.527250 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.527256 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.527261 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.527267 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527273 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527290 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527296 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.527302 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.527308 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.527314 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.527320 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.527326 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.527332 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.527338 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.527344 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.527350 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.527355 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.527361 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.527370 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.527376 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.527385 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.527391 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.527397 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.527405 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.527411 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.527417 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.527423 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.527429 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.527435 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.527441 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.527447 | controller | 18:52:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.527453 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.527458 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.527464 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.527470 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.527476 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.527482 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.527488 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.527494 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.527500 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.527505 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.527511 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.527517 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.527523 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527529 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527535 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527550 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.527556 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.527562 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.527568 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.527574 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.527586 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.527593 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.527599 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.527604 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.527610 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.527616 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.527622 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.527628 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.527634 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.527639 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.527645 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.527651 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.527657 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.527663 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.527669 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.527675 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.527681 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.527687 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.527693 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.527699 | controller | 18:52:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.527704 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.527710 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.527716 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.527724 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.527730 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.527736 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.527742 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.527748 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.527754 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.527760 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.527766 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.527772 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.527778 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527786 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527792 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.527798 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.527804 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.527819 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.527826 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.527832 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.527838 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.527844 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.527850 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.527856 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.527861 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.527867 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.527873 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.527879 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.527885 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.527891 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.527911 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.527917 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.527923 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.527929 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.527936 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.527942 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.527948 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.527953 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.527959 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.527965 | controller | 18:52:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.527971 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.527977 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.527983 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.527989 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.528000 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.528007 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.528012 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.528018 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.528024 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.528030 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.528034 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.528039 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.528043 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528047 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528051 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528056 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.528060 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.528064 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.528078 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.528082 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528087 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528092 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.528099 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.528108 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.528114 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.528120 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.528126 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.528132 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.528138 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.528158 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.528169 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.528175 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528181 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.528187 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528193 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.528199 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.528205 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.528214 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.528220 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.528226 | controller | 18:52:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.528232 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.528238 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.528244 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.528250 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.528256 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.528262 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.528267 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.528273 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.528279 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.528285 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.528291 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.528297 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.528303 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528309 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528314 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528320 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.528326 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.528332 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.528338 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.528353 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528360 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528366 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.528372 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.528378 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.528384 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.528388 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.528393 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.528397 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.528401 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.528408 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.528412 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.528417 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528421 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.528425 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528430 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.528434 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.528438 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.528442 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.528447 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.528451 | controller | 18:52:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.528455 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.528460 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.528466 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.528471 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.528475 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.528479 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.528484 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.528488 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.528492 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.528496 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.528501 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.528505 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.528509 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528514 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528518 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528525 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.528529 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.528534 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.528538 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.528542 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528556 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528561 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.528565 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.528570 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.528574 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.528578 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.528582 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.528587 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.528591 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.528595 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.528600 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.528604 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528608 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.528613 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528617 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.528621 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.528625 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.528630 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.528634 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.528638 | controller | 18:52:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.528643 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.528647 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.528651 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.528656 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.528660 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.528664 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.528668 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.528673 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.528677 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.528681 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.528686 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.528690 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.528694 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528700 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528704 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528709 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.528713 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.528717 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.528722 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.528726 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528732 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528743 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.528755 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.528760 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.528764 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.528768 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.528772 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.528777 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.528781 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.528785 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.528790 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.528794 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528798 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.528803 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528807 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.528811 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.528816 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.528821 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.528827 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.528834 | controller | 18:52:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.528840 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.528846 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.528852 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.528857 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.528863 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.528868 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.528872 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.528876 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.528881 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.528885 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.528889 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.528894 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.528898 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528902 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528906 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.528911 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.528915 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.528919 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.528924 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.528928 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528932 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.528937 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.528941 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.528953 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.528958 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.528963 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.528967 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.528971 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.528976 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.528980 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.528986 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.528991 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.528995 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.528999 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529004 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.529008 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.529012 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.529018 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529023 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.529027 | controller | 18:52:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.529031 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529037 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.529042 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.529046 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.529050 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.529054 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.529059 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.529063 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.529067 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.529071 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.529076 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.529080 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.529084 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529089 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529093 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529097 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.529101 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.529106 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.529110 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.529114 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529139 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529162 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.529171 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.529175 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.529179 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.529190 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.529195 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.529199 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.529204 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.529210 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.529215 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.529219 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529223 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.529228 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529232 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.529236 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.529241 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.529245 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529249 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.529254 | controller | 18:52:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.529258 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529262 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.529267 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.529271 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.529275 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.529279 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.529284 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.529288 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.529292 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.529297 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.529301 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.529305 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.529309 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529314 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529318 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529322 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.529329 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.529333 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.529338 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.529342 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529348 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529352 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.529357 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.529361 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.529365 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.529370 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.529380 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.529385 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.529389 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.529394 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.529398 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.529402 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529407 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.529411 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529415 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.529420 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.529424 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.529428 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529433 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.529437 | controller | 18:52:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.529442 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529446 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.529450 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.529454 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.529460 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.529465 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.529469 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.529474 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.529478 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.529482 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.529487 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.529491 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.529495 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529501 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529506 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529510 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.529514 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.529519 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.529523 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.529527 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529532 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529536 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.529540 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.529545 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.529549 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.529553 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.529558 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.529562 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.529572 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.529577 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.529581 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.529586 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529590 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.529594 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529599 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.529603 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.529607 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.529612 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529616 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.529620 | controller | 18:52:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.529627 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529631 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.529636 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.529640 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.529647 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.529652 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.529656 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.529660 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.529665 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.529669 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.529674 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.529678 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.529682 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529686 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529691 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529695 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.529699 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.529704 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.529708 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.529712 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529718 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529723 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.529727 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.529731 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.529736 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.529740 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.529744 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.529748 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.529753 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.529763 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.529768 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.529772 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529777 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.529781 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.529785 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.529790 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.529794 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.529800 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529806 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.529810 | controller | 18:52:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.529816 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.529822 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.529828 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.529834 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.529840 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.529846 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.529852 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.529858 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.529864 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.529869 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.529873 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.529877 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.529882 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529886 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529893 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.529898 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.529903 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.529907 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.529911 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.529916 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529921 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.529928 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.529934 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.529940 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.529946 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.529952 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.529958 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.529964 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.529970 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.529979 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.529992 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.529996 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530001 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.530005 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530010 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.530014 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.530021 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.530027 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530033 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.530039 | controller | 18:52:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.530046 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530052 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.530058 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.530064 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.530069 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.530076 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.530081 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.530085 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.530090 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.530094 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.530098 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.530103 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.530107 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530111 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530117 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530123 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.530129 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.530135 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.530142 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.530166 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.530175 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.530181 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.530187 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.530193 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.530199 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.530203 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.530207 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.530212 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.530218 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.530225 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.530231 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.530237 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530253 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.530260 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530266 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.530272 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.530278 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.530284 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530293 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.530299 | controller | 18:52:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.530305 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530311 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.530317 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.530323 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.530329 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.530335 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.530341 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.530347 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.530353 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.530359 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.530364 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.530370 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.530376 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530385 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530391 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530397 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.530403 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.530409 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.530415 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.530421 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.530427 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.530433 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.530438 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.530444 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.530450 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.530456 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.530462 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.530468 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.530473 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.530479 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.530488 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.530495 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530501 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.530516 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530522 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.530528 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.530534 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.530540 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530546 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.530552 | controller | 18:52:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.530558 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530564 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.530570 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.530576 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.530585 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.530592 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.530597 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.530603 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.530609 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.530615 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.530621 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.530627 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.530633 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530641 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530647 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530653 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.530659 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.530665 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.530671 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.530677 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.530683 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.530689 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.530695 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.530701 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.530706 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.530712 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.530718 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.530724 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.530730 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.530736 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.530742 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.530747 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530753 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.530759 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530775 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.530781 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.530787 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.530796 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530802 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.530808 | controller | 18:52:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.530814 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530820 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.530826 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.530832 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.530837 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.530841 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.530846 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.530850 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.530854 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.530858 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.530863 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.530867 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.530871 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530875 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530880 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.530884 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.530888 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.530892 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.530897 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.530901 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.530907 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.530912 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.530916 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.530920 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.530925 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.530929 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.530933 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.530937 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.530942 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.530949 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.530954 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.530958 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530962 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.530967 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.530971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.530977 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.530988 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.530993 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.530997 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.531002 | controller | 18:52:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.531006 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531010 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.531014 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.531019 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.531025 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.531029 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.531033 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.531038 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.531042 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.531046 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.531050 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.531055 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.531059 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531063 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531067 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531072 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.531078 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.531084 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.531090 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.531096 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.531105 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.531111 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.531117 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.531122 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.531128 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.531133 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.531139 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.531159 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.531169 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.531175 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.531179 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.531183 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.531188 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.531192 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.531196 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.531200 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.531205 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.531217 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531222 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.531226 | controller | 18:52:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.531231 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531235 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.531239 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.531243 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.531247 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.531252 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.531256 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.531260 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.531265 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.531269 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.531273 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.531277 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.531284 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531291 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531295 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531299 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.531303 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.531308 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.531312 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.531316 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.531320 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.531325 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.531330 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.531336 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.531342 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.531348 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.531354 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.531360 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.531366 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.531372 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.531378 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.531383 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.531388 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.531392 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.531396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.531400 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.531405 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.531409 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531421 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.531425 | controller | 18:52:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.531430 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531436 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.531442 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.531447 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.531456 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.531463 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.531468 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.531474 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.531480 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.531486 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.531492 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.531496 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.531500 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531505 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531509 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531516 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.531520 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.531525 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.531530 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.531536 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.531542 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.531548 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.531554 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.531560 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.531566 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.531572 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.531578 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.531583 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.531589 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.531595 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.531601 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.531607 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.531614 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.531618 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.531623 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.531628 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.531634 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.531644 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531650 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.531655 | controller | 18:52:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.531674 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531681 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.531687 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.531692 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.531698 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.531704 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.531710 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.531716 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.531722 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.531728 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.531734 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.531740 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.531746 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531752 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531758 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.531764 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.531769 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.531775 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.531781 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.531787 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.531793 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.531799 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.531805 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.531811 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.531817 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.531823 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.531828 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.531834 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.531840 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.531849 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.531855 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.531861 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.531867 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.531873 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.531879 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.531884 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.531910 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.531917 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531923 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.531929 | controller | 18:52:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.531935 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.531950 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.531956 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.531962 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.531968 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.531974 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.531980 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.531986 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.531992 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.531997 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.532003 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.532013 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.532019 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532025 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532031 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532037 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.532043 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.532049 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.532055 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.532061 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.532074 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.532080 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.532087 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.532093 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.532100 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.532107 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.532113 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.532121 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.532127 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.532133 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.532141 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.532165 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.532171 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.532181 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.532188 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.532194 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.532199 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.532205 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.532211 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.532217 | controller | 18:52:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.532223 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.532229 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.532245 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.532251 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.532257 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.532263 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.532269 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.532275 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.532281 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.532286 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.532291 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.532296 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.532300 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532307 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532311 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532316 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.532320 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.532324 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.532328 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.532333 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.532337 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.532341 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.532345 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.532350 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.532354 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.532358 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.532363 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.532367 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.532373 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.532378 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.532383 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.532388 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.532395 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.532401 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.532407 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.532413 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.532418 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.532424 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.532430 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.532436 | controller | 18:52:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.532442 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.532448 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.532454 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.532460 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.532478 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.532487 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.532494 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.532500 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.532506 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.532511 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.532517 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.532523 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.532529 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532535 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532541 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532547 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.532553 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.532558 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.532564 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.532570 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.532576 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.532582 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.532588 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.532594 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.532600 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.532606 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.532612 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.532618 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.532624 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.532629 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.532635 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.532641 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.532647 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.532653 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.532659 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.532665 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.532671 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.532680 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.532686 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.532692 | controller | 18:52:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.532698 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.532704 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.532709 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.532715 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.532724 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.532739 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.532745 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.532751 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.532758 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.532763 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.532772 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.532778 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.532784 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532790 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532796 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.532802 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.532807 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.532813 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.532819 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.532825 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.532831 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.532837 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.532843 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.532849 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.532855 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.532861 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.532866 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.532872 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.532877 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.532886 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.532892 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.532898 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.532904 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.532910 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.532916 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.532922 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.532928 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.532933 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.532939 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.532945 | controller | 18:52:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.532951 | controller | \"ac-watcher\" not found\n logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.532957 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.532963 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.532969 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.532975 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.532981 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.532986 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.533002 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.533009 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.533015 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.533020 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.533026 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.533032 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533038 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533044 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533050 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.533056 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.533062 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.533068 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.533074 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533083 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533089 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.533095 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.533101 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.533107 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.533112 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.533118 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.533127 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.533133 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.533139 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.533166 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.533176 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.533182 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.533188 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.533194 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.533199 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.533205 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.533211 | controller | expected application_credential_id\"]\n logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.533217 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.533223 | controller | 18:52:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.533229 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.533235 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.533240 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.533246 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.533252 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.533258 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.533264 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.533270 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.533285 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.533291 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.533297 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.533303 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.533310 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533318 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533322 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533329 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.533333 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.533338 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.533342 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.533346 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533352 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533357 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.533361 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.533365 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.533370 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.533374 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.533378 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.533384 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.533390 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.533397 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.533403 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.533409 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.533415 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.533421 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.533427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.533433 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.533439 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.533445 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.533451 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.533457 | controller | 18:53:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.533463 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.533469 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.533475 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.533481 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.533490 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.533495 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.533499 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.533504 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.533508 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.533526 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.533532 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.533536 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.533540 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533545 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533549 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533553 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.533558 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.533562 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.533566 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.533571 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533575 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533579 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.533583 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.533588 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.533592 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.533596 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.533601 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.533605 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.533609 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.533613 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.533618 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.533622 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.533626 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.533631 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.533635 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.533639 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.533644 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.533650 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.533654 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.533658 | controller | 18:53:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.533663 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.533667 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.533671 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.533675 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.533680 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.533684 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.533688 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.533693 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.533697 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.533701 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.533705 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.533717 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.533721 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533726 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533730 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533734 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.533739 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.533743 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.533747 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.533752 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533757 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533763 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.533769 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.533775 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.533781 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.533790 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.533796 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.533803 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.533807 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.533813 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.533819 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.533824 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.533828 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.533833 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.533837 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.533841 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.533845 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.533850 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.533854 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.533859 | controller | 18:53:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.533863 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.533867 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.533871 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.533876 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.533880 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.533884 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.533889 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.533893 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.533897 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.533901 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.533906 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.533910 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.533922 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533927 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533931 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.533936 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.533940 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.533945 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.533951 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.533957 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533966 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.533972 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.533978 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.533984 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.533990 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.533996 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.534002 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.534006 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.534011 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.534015 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.534019 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.534023 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534028 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.534034 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534040 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.534047 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.534053 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.534059 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.534065 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.534070 | controller | 18:53:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.534076 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.534082 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.534088 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.534094 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.534098 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.534103 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.534107 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.534111 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.534118 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.534123 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.534127 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.534131 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.534135 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534142 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534172 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534177 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.534182 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.534186 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.534191 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.534195 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.534199 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.534204 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.534208 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.534212 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.534217 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.534221 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.534225 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.534229 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.534234 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.534238 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.534242 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.534247 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534251 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.534255 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534260 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.534264 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.534268 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.534273 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.534278 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.534284 | controller | 18:53:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.534290 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.534296 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.534302 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.534309 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.534321 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.534328 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.534334 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.534340 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.534346 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.534351 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.534356 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.534360 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.534364 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534369 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534373 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534386 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.534390 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.534395 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.534399 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.534405 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.534414 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.534420 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.534427 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.534433 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.534441 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.534447 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.534453 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.534459 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.534463 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.534468 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.534472 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.534476 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534481 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.534485 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534489 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.534494 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.534498 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.534506 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.534512 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.534519 | controller | 18:53:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.534525 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.534531 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.534537 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.534543 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.534549 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.534555 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.534561 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.534566 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.534570 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.534575 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.534579 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.534583 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.534588 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534592 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534596 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534602 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.534608 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.534624 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.534632 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.534638 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.534644 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.534650 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.534656 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.534662 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.534668 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.534674 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.534679 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.534684 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.534688 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.534696 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.534703 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.534709 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534715 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.534721 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534727 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.534733 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.534739 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.534745 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.534751 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.534757 | controller | 18:53:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.534763 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.534769 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.534775 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.534783 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.534790 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.534796 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.534802 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.534808 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.534814 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.534820 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.534826 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.534832 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.534838 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534844 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534850 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.534860 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.534866 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.534872 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.534888 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.534895 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.534905 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.534911 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.534917 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.534923 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.534929 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.534935 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.534941 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.534946 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.534952 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.534958 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.534964 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.534970 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534976 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.534982 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.534989 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.534995 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.535000 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.535006 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.535012 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.535018 | controller | 18:53:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.535024 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.535030 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.535036 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.535042 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.535048 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.535054 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.535060 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.535066 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.535072 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.535077 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.535082 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.535086 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.535091 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535097 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535102 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535106 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.535110 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.535115 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.535119 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.535131 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.535136 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.535140 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.535158 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.535168 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.535173 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.535177 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.535181 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.535186 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.535190 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.535194 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.535199 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.535205 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.535211 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.535220 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.535227 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.535233 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.535239 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.535245 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.535252 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.535257 | controller | 18:53:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.535262 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.535268 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.535274 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.535280 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.535289 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.535295 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.535301 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.535307 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.535312 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.535317 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.535321 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.535325 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.535330 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535334 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535338 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535342 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.535347 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.535351 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.535356 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.535362 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.535379 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.535387 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.535393 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.535399 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.535405 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.535411 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.535416 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.535421 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.535425 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.535429 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.535437 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.535442 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.535447 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.535452 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.535459 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.535465 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.535471 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.535481 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.535487 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.535493 | controller | 18:53:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.535499 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.535505 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.535513 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.535519 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.535525 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.535531 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.535535 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.535539 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.535544 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.535550 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.535556 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.535562 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.535568 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535574 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535580 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535586 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.535592 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.535598 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.535604 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.535610 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.535619 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.535625 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.535641 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.535648 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.535654 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.535660 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.535666 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.535672 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.535678 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.535687 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.535693 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.535698 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.535704 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.535710 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.535716 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.535722 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.535728 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.535734 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.535740 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.535746 | controller | 18:53:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.535752 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.535758 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.535764 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.535770 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.535776 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.535782 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.535788 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.535793 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.535799 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.535805 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.535811 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.535817 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.535823 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535829 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535835 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.535841 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.535847 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.535853 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.535859 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.535865 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.535873 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.535882 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.535931 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.535950 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.535958 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.535964 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.535971 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.535977 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.535982 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.535988 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.535994 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.536000 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536006 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.536012 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536018 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.536024 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.536030 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.536035 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536041 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.536047 | controller | 18:53:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.536053 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536059 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.536065 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.536071 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.536082 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.536089 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.536094 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.536100 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.536106 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.536112 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.536118 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.536124 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.536130 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536139 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536165 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536173 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.536180 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.536186 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.536192 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.536198 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.536205 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.536211 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.536217 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.536223 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.536229 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.536246 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.536253 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.536259 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.536265 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.536271 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.536276 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.536282 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536288 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.536294 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536300 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.536306 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.536312 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.536317 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536321 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.536325 | controller | 18:53:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.536330 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536337 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.536342 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.536346 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.536353 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.536357 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.536361 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.536366 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.536370 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.536374 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.536378 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.536383 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.536387 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536391 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536395 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536400 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.536404 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.536408 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.536413 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.536417 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.536421 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.536425 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.536430 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.536434 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.536438 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.536443 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.536455 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.536462 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.536468 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.536474 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.536481 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.536486 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536491 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.536495 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536500 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.536504 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.536508 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.536515 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536519 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.536524 | controller | 18:53:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.536528 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536532 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.536537 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.536541 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.536545 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.536549 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.536554 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.536558 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.536562 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.536567 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.536571 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.536575 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.536580 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536584 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536588 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536595 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.536600 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.536605 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.536609 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.536613 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.536617 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.536622 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.536626 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.536630 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.536635 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.536639 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.536644 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.536650 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.536665 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.536675 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.536682 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.536688 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536695 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.536701 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536707 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.536712 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.536716 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.536721 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536725 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.536729 | controller | 18:53:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.536734 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536738 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.536744 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.536750 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.536756 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.536762 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.536768 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.536774 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.536780 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.536786 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.536792 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.536799 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.536805 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536811 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536817 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.536823 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.536829 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.536835 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.536841 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.536847 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.536860 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.536866 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.536872 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.536876 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.536880 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.536885 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.536891 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.536897 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.536903 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.536921 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.536928 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.536934 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536940 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.536946 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.536951 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.536956 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.536960 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.536964 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536969 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.536974 | controller | 18:53:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.536983 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.536990 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.536996 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.537002 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.537008 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.537014 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.537020 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.537026 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.537032 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.537039 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.537045 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.537052 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.537058 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537068 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537074 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537080 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.537086 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.537092 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.537098 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.537105 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.537111 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.537117 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.537123 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.537129 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.537135 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.537141 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.537170 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.537177 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.537183 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.537189 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.537208 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.537216 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.537222 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.537228 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.537234 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.537240 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.537246 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.537252 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.537257 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.537263 | controller | 18:53:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.537267 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.537271 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.537277 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.537283 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.537293 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.537300 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.537306 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.537312 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.537317 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.537323 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.537329 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.537335 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.537341 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537346 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537355 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537361 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.537367 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.537373 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.537379 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.537385 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.537391 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.537397 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.537403 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.537409 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.537414 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.537420 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.537426 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.537432 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.537438 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.537444 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.537449 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.537455 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.537472 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.537479 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.537485 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.537491 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.537496 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.537507 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.537513 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.537519 | controller | 18:53:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.537525 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.537531 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.537537 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.537543 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.537548 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.537554 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.537560 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.537564 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.537568 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.537573 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.537577 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.537581 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.537585 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537590 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537594 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537598 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.537602 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.537606 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.537611 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.537615 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.537620 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.537626 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.537632 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.537638 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.537644 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.537649 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.537655 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.537661 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.537667 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.537676 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.537682 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.537687 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.537693 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.537709 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.537715 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.537721 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.537727 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.537733 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.537742 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.537748 | controller | 18:53:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.537754 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.537760 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.537766 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.537772 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.537779 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.537785 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.537791 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.537797 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.537803 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.537809 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.537815 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.537820 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.537826 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537832 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537838 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.537844 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.537850 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.537856 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.537861 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.537867 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.537875 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.537881 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.537887 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.537893 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.537899 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.537905 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.537911 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.537916 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.537922 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.537928 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.537934 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.537940 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.537946 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.537952 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.537968 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.537974 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.537980 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.537986 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.537992 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.537998 | controller | 18:53:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.538003 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.538009 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.538015 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.538021 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.538026 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.538036 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.538042 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.538048 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.538054 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.538060 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.538066 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.538071 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.538077 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538086 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538092 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538099 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.538109 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.538115 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.538122 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.538129 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.538135 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.538143 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.538167 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.538173 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.538179 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.538184 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.538190 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.538196 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.538202 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.538207 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.538213 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.538219 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.538225 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.538231 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.538237 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.538243 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.538259 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.538265 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.538271 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.538277 | controller | 18:53:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.538283 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.538289 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.538295 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.538301 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.538310 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.538316 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.538322 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.538328 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.538334 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.538340 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.538345 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.538351 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.538357 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538363 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538369 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538378 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.538384 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.538390 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.538396 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.538402 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.538407 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.538413 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.538419 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.538425 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.538431 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.538437 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.538442 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.538448 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.538454 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.538460 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.538466 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.538472 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.538478 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.538484 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.538490 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.538498 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.538504 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.538522 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.538528 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.538534 | controller | 18:53:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.538541 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.538546 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.538552 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.538558 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.538564 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.538570 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.538576 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.538581 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.538587 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.538593 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.538599 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.538605 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.538611 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538617 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538623 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538628 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.538634 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.538640 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.538646 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.538652 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.538658 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.538663 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.538669 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.538675 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.538681 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.538687 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.538692 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.538698 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.538704 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.538713 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.538719 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.538725 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.538731 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.538737 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.538743 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.538749 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.538754 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.538760 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.538776 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.538782 | controller | 18:53:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.538788 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.538794 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.538800 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.538806 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.538812 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.538818 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.538824 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.538830 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.538836 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.538841 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.538847 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.538853 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.538862 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538868 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538874 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.538880 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.538885 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.538896 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.538902 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.538908 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.538917 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.538923 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.538928 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.538934 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.538940 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.538946 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.538952 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.538958 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.538963 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.538969 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.538977 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.538983 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.538989 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.538995 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.539000 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.539006 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.539012 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.539018 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.539024 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.539039 | controller | 18:53:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.539046 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.539052 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.539058 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.539064 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.539070 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.539076 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.539082 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.539088 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.539094 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.539099 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.539105 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.539111 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.539117 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539126 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539132 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539138 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.539157 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.539167 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.539173 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.539179 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.539185 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.539190 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.539196 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.539202 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.539208 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.539214 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.539220 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.539226 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.539232 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.539238 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.539244 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.539250 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.539259 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.539265 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.539271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.539276 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.539282 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.539288 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.539294 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.539300 | controller | 18:53:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.539306 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.539322 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.539329 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.539335 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.539344 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.539350 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.539355 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.539361 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.539367 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.539373 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.539379 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.539384 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.539390 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539396 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539402 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539408 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.539414 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.539420 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.539425 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.539431 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.539439 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.539445 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.539451 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.539457 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.539463 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.539469 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.539475 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.539481 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.539486 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.539493 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.539499 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.539504 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.539511 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.539517 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.539523 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.539529 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.539535 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.539545 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.539551 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.539557 | controller | 18:53:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.539563 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.539569 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.539586 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.539594 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.539603 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.539610 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.539616 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.539622 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.539628 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.539634 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.539640 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.539653 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.539659 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539665 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539671 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539677 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.539683 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.539689 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.539694 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.539700 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.539706 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.539712 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.539718 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.539724 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.539729 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.539735 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.539741 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.539747 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.539753 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.539762 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.539767 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.539773 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.539779 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.539785 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.539791 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.539797 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.539803 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.539809 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.539815 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.539821 | controller | 18:53:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.539827 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.539832 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.539836 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.539840 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.539854 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.539859 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.539863 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.539867 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.539871 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.539876 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.539899 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.539904 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.539909 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539913 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539917 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.539921 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.539926 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.539930 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.539934 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.539938 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.539945 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.539949 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.539954 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.539958 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.539962 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.539966 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.539971 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.539975 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.539983 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.539987 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.539991 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.539996 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540000 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.540004 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540008 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.540012 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.540017 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.540021 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540025 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.540029 | controller | 18:53:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.540033 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540038 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.540042 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.540046 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.540050 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.540061 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.540066 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.540070 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.540074 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.540079 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.540083 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.540087 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.540091 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540097 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540102 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540107 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.540112 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.540116 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.540120 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.540124 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540129 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540133 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.540137 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.540141 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.540145 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.540165 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.540169 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.540174 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.540178 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.540182 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.540186 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.540190 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540195 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.540199 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540203 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.540207 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.540211 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.540216 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540220 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.540224 | controller | 18:53:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.540229 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540233 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.540237 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.540241 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.540249 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.540254 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.540265 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.540270 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.540274 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.540278 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.540285 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.540289 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.540293 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540297 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540302 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540306 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.540310 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.540314 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.540318 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.540323 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540327 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540331 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.540335 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.540339 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.540343 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.540349 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.540355 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.540361 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.540367 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.540373 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.540378 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.540383 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540387 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.540391 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540395 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.540399 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.540404 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.540410 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540414 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.540419 | controller | 18:53:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.540423 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540427 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.540431 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.540435 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.540440 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.540444 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.540448 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.540452 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.540464 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.540469 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.540473 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.540477 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.540482 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540486 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540490 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540494 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.540499 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.540503 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.540507 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.540511 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540518 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540522 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.540526 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.540530 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.540535 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.540539 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.540543 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.540549 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.540553 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.540559 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.540565 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.540569 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540573 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.540577 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540582 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.540586 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.540590 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.540594 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540599 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.540603 | controller | 18:53:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.540607 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540611 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.540615 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.540620 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.540624 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.540628 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.540632 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.540637 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.540641 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.540652 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.540656 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.540661 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.540665 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540669 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540673 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540678 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.540682 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.540686 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.540690 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.540694 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540701 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540706 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.540712 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.540718 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.540724 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.540730 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.540736 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.540742 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.540748 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.540754 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.540758 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.540762 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540767 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.540771 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540775 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.540779 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.540783 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.540788 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540794 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.540799 | controller | 18:53:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.540803 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.540807 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.540813 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.540819 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.540825 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.540831 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.540837 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.540843 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.540849 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.540858 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.540863 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.540876 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.540880 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540887 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540892 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.540896 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.540900 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.540905 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.540911 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.540917 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540923 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.540929 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.540935 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.540941 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.540947 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.540952 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.540958 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.540964 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.540969 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.540973 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.540977 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.540982 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540986 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.540990 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.540994 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.540999 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.541004 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.541010 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541016 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.541022 | controller | 18:53:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.541028 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541034 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.541040 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.541046 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.541057 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.541063 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.541069 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.541075 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.541081 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.541085 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.541089 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.541094 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.541111 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541119 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541127 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541133 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.541139 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.541145 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.541172 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.541178 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.541184 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.541190 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.541196 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.541201 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.541207 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.541216 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.541222 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.541228 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.541234 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.541240 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.541246 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.541251 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.541257 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.541263 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.541269 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.541275 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.541281 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.541289 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541296 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.541302 | controller | 18:53:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.541307 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541313 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.541319 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.541325 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.541331 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.541337 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.541343 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.541348 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.541354 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.541360 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.541365 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.541371 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.541377 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541383 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541399 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541406 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.541412 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.541418 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.541424 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.541430 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.541436 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.541442 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.541447 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.541453 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.541459 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.541465 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.541471 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.541477 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.541483 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.541491 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.541496 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.541500 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.541504 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.541508 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.541513 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.541517 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.541521 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.541525 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541530 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.541534 | controller | 18:53:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.541538 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541542 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.541547 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.541551 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.541555 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.541559 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.541563 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.541568 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.541574 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.541578 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.541583 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.541587 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.541591 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541595 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541599 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541613 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.541618 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.541622 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.541626 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.541630 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.541638 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.541642 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.541646 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.541650 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.541655 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.541661 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.541667 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.541673 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.541679 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.541685 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.541691 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.541697 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.541701 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.541706 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.541710 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.541714 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.541718 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.541723 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541727 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.541731 | controller | 18:53:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.541735 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541740 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.541744 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.541748 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.541752 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.541756 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.541761 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.541766 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.541772 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.541778 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.541784 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.541789 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.541795 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541804 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541810 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.541816 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.541820 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.541832 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.541837 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.541841 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.541845 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.541849 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.541854 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.541858 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.541866 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.541873 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.541879 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.541885 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.541891 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.541897 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.541902 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.541908 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.541914 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.541920 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.541924 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.541929 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.541933 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.541937 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541941 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.541946 | controller | 18:53:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.541950 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.541954 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.541960 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.541966 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.541975 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.541981 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.541987 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.541993 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.541999 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.542005 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.542010 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.542016 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.542022 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542028 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542034 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542038 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.542042 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.542047 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.542062 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.542069 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542075 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542081 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.542087 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.542093 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.542099 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.542104 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.542110 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.542116 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.542122 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.542128 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.542137 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.542143 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542162 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.542171 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542177 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.542183 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.542189 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.542198 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.542204 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.542210 | controller | 18:53:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.542216 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.542222 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.542227 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.542236 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.542242 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.542248 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.542253 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.542259 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.542265 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.542271 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.542277 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.542283 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.542289 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542295 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542301 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542307 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.542312 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.542318 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.542324 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.542340 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542346 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542352 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.542358 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.542364 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.542369 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.542373 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.542378 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.542382 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.542386 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.542393 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.542397 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.542401 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542406 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.542410 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542414 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.542418 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.542422 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.542427 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.542431 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.542435 | controller | 18:53:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.542439 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.542444 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.542448 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.542452 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.542456 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.542461 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.542465 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.542469 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.542473 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.542478 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.542482 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.542486 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.542490 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542494 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542499 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542503 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.542507 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.542511 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.542516 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.542520 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542532 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542537 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.542541 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.542547 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.542551 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.542556 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.542560 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.542564 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.542568 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.542572 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.542577 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.542581 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542585 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.542589 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542594 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.542598 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.542602 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.542608 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.542612 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.542617 | controller | 18:53:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.542621 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.542625 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.542629 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.542634 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.542641 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.542645 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.542649 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.542654 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.542658 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.542662 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.542666 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.542671 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.542675 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542725 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542730 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542734 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.542739 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.542743 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.542747 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.542752 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542758 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542762 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.542774 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.542779 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.542783 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.542787 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.542791 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.542796 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.542800 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.542804 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.542808 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.542813 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542817 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.542821 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542826 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.542830 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.542834 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.542838 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.542843 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.542847 | controller | 18:53:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.542851 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.542855 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.542861 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.542866 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.542872 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.542876 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.542880 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.542885 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.542889 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.542893 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.542897 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.542902 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.542906 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542910 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542914 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.542918 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.542923 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.542927 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.542931 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.542936 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542940 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.542944 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.542948 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.542959 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.542963 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.542968 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.542972 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.542976 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.542980 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.542985 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.542989 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.542993 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.542997 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.543002 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543006 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.543010 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.543014 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.543020 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543024 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.543029 | controller | 18:53:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.543033 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543037 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.543041 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.543045 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.543050 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.543054 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.543058 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.543062 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.543067 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.543071 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.543075 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.543079 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.543084 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543088 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543092 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543098 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.543102 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.543106 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.543111 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.543115 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543119 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543125 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.543129 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.543134 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.543144 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.543163 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.543167 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.543172 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.543176 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.543182 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.543187 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.543191 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543195 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.543199 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543204 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.543209 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.543215 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.543221 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543227 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.543233 | controller | 18:53:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.543239 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543244 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.543248 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.543252 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.543257 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.543263 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.543268 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.543272 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.543276 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.543280 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.543285 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.543289 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.543293 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543297 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543301 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543306 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.543310 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.543314 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.543318 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.543323 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543329 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543333 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.543337 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.543341 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.543346 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.543350 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.543362 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.543367 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.543371 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.543375 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.543379 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.543384 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543388 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.543392 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.543401 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.543405 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.543409 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543413 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.543417 | controller | 18:53:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.543422 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543428 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.543432 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.543436 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.543440 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.543445 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.543449 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.543453 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.543457 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.543462 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.543466 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.543470 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.543474 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543481 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543485 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543489 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.543494 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.543498 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.543502 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.543506 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543511 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543515 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.543519 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.543523 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.543528 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.543532 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.543536 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.543546 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.543551 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.543555 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.543561 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.543566 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543570 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.543574 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543578 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.543582 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.543587 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.543591 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543595 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.543599 | controller | 18:53:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.543603 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543608 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.543612 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.543616 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.543622 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.543626 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.543630 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.543635 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.543639 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.543643 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.543647 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.543651 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.543656 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543660 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543664 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543668 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.543675 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.543680 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.543684 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.543688 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543694 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543698 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.543702 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.543708 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.543714 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.543719 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.543724 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.543729 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.543735 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.543748 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.543754 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.543760 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543766 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.543772 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.543777 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.543782 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.543787 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.543796 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543802 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.543808 | controller | 18:53:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.543814 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.543820 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.543826 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.543832 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.543838 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.543842 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.543846 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.543851 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.543855 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.543859 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.543863 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.543868 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.543872 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543892 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543898 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.543904 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.543909 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.543914 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.543920 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.543925 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543930 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.543936 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.543941 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.543947 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.543952 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.543958 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.543964 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.543971 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.543977 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.543987 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.544002 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.544007 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.544011 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.544016 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.544020 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.544024 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.544028 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.544033 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.544037 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.544042 | controller | 18:53:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.544051 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.544057 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.544063 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.544069 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.544077 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.544083 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.544089 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.544095 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.544103 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.544109 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.544115 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.544121 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.544126 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544130 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544134 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544139 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.544158 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.544169 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.544176 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.544181 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.544191 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.544197 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.544233 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.544242 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.544248 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.544254 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.544260 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.544266 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.544272 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.544278 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.544284 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.544300 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.544307 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.544312 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.544318 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.544324 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.544330 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.544336 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.544342 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.544348 | controller | 18:53:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.544354 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.544360 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.544366 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.544372 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.544378 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.544384 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.544390 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.544395 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.544401 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.544407 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.544413 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.544419 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.544425 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544434 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544443 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544449 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.544455 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.544461 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.544467 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.544473 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.544479 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.544485 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.544491 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.544496 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.544502 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.544508 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.544514 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.544520 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.544526 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.544532 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.544538 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.544544 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.544550 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.544565 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.544571 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.544577 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.544583 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.544589 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.544594 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.544600 | controller | 18:53:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.544606 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.544612 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.544618 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.544624 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.544633 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.544639 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.544645 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.544651 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.544657 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.544662 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.544668 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.544674 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.544680 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544686 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544692 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544701 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.544707 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.544713 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.544719 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.544725 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.544730 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.544736 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.544742 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.544748 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.544754 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.544760 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.544765 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.544771 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.544777 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.544783 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.544789 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.544795 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.544801 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.544807 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.544821 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.544828 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.544834 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.544842 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.544851 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.544857 | controller | 18:53:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.544863 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.544869 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.544875 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.544880 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.544886 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.544892 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.544898 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.544904 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.544910 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.544915 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.544921 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.544927 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.544933 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544939 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544945 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.544951 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.544956 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.544962 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.544968 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.544974 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.544982 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.544988 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.544994 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.545000 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.545006 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.545012 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.545018 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.545023 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.545029 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.545038 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.545044 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.545050 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.545056 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.545062 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.545068 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.545083 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.545089 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.545095 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.545101 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.545107 | controller | 18:53:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.545113 | controller | \"ac-watcher\" not found\n logger.go:42: 18:53:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.545119 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.545125 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.545131 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.545137 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.545142 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.545260 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.545269 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.545275 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.545281 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.545286 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.545292 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.545298 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545310 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545317 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545322 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.545328 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.545334 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.545340 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.545346 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.545355 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.545361 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.545366 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.545372 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.545378 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.545384 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.545390 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.545395 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.545401 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.545407 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.545414 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.545419 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.545424 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.545430 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.545436 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.545442 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.545448 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.545464 | controller | expected application_credential_id\"]\n logger.go:42: 18:53:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.545472 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.545478 | controller | 18:53:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.545484 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.545490 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.545496 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.545502 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.545507 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.545513 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.545519 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.545525 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.545531 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.545537 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.545543 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.545549 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.545555 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545564 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545570 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545576 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.545582 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.545588 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.545593 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.545599 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.545605 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.545611 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.545617 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.545623 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.545629 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.545635 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.545641 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.545647 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.545653 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.545659 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.545665 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.545670 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.545676 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.545682 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.545688 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.545697 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.545703 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.545709 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.545725 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.545732 | controller | 18:54:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.545738 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.545744 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.545750 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.545756 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.545765 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.545771 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.545777 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.545783 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.545789 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.545795 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.545801 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.545807 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.545813 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545819 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545825 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.545831 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.545837 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.545843 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.545848 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.545854 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.545860 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.545866 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.545872 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.545878 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.545884 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.545890 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.545896 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.545902 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.545908 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.545914 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.545919 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.545925 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.545931 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.545937 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.545943 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.545949 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.545955 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.545963 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.545970 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.545988 | controller | 18:54:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.545996 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546002 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.546008 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.546014 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.546020 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.546025 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.546031 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.546037 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.546043 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.546049 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.546055 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.546061 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.546070 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546076 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546082 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546088 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.546094 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.546100 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.546106 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.546111 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546117 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546123 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.546129 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.546135 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.546141 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.546161 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.546172 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.546178 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.546184 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.546193 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.546199 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.546205 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.546211 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.546216 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.546222 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.546228 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.546234 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.546240 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546248 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.546254 | controller | 18:54:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.546260 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546276 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.546283 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.546289 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.546295 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.546301 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.546307 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.546313 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.546319 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.546325 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.546331 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.546336 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.546342 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546348 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546354 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546360 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.546366 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.546372 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.546378 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.546383 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546395 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546401 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.546407 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.546413 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.546419 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.546425 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.546430 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.546436 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.546442 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.546448 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.546454 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.546460 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.546468 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.546473 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.546478 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.546482 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.546486 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.546491 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546495 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.546499 | controller | 18:54:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.546504 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546508 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.546520 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.546525 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.546529 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.546533 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.546538 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.546542 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.546546 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.546550 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.546555 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.546559 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.546563 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546570 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546574 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546582 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.546586 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.546590 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.546595 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.546599 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546603 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546608 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.546612 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.546616 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.546620 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.546625 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.546629 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.546633 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.546637 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.546642 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.546646 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.546650 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.546655 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.546659 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.546663 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.546668 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.546672 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.546676 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546680 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.546685 | controller | 18:54:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.546689 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546693 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.546698 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.546702 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.546715 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.546719 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.546724 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.546728 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.546732 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.546737 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.546741 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.546747 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.546751 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546756 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546760 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546764 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.546769 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.546773 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.546777 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.546781 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546786 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546790 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.546794 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.546799 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.546803 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.546807 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.546811 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.546816 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.546820 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.546824 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.546829 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.546833 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.546837 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.546842 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.546846 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.546850 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.546854 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.546860 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546865 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.546869 | controller | 18:54:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.546873 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.546879 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.546885 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.546891 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.546897 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.546912 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.546917 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.546922 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.546926 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.546930 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.546935 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.546939 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.546943 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546948 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546952 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.546956 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.546960 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.546967 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.546972 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.546976 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546980 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.546985 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.546989 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.546993 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.546998 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.547002 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.547006 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.547011 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.547016 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.547023 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.547029 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.547033 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.547037 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.547042 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.547046 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.547050 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.547055 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.547059 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.547063 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.547067 | controller | 18:54:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.547072 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.547076 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.547080 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.547085 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.547089 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.547093 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.547105 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.547110 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.547114 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.547119 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.547125 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.547131 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.547137 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547143 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547167 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547172 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.547176 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.547181 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.547185 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.547189 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.547196 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.547200 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.547205 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.547209 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.547213 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.547218 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.547223 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.547229 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.547235 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.547241 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.547247 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.547253 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.547259 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.547265 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.547271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.547277 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.547283 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.547288 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.547294 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.547300 | controller | 18:54:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.547304 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.547308 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.547313 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.547317 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.547322 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.547328 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.547334 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.547340 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.547357 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.547364 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.547373 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.547379 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.547384 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547390 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547395 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547399 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.547403 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.547408 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.547412 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.547418 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.547427 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.547433 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.547439 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.547445 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.547451 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.547457 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.547463 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.547469 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.547475 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.547481 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.547487 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.547493 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.547497 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.547501 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.547506 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.547511 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.547518 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.547524 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.547530 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.547539 | controller | 18:54:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.547545 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.547551 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.547557 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.547563 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.547573 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.547579 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.547585 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.547591 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.547597 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.547612 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.547620 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.547626 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.547631 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547637 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547643 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547649 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.547655 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.547661 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.547667 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.547672 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.547678 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.547684 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.547690 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.547696 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.547702 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.547708 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.547713 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.547724 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.547730 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.547736 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.547742 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.547748 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.547753 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.547759 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.547765 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.547771 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.547777 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.547786 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.547792 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.547798 | controller | 18:54:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.547804 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.547810 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.547815 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.547821 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.547827 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.547833 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.547839 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.547845 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.547850 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.547856 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.547862 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.547891 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.547897 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547902 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547906 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.547910 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.547915 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.547921 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.547927 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.547933 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.547939 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.547945 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.547951 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.547956 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.547962 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.547968 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.547974 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.547980 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.547986 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.547995 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.548001 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.548007 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.548012 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.548018 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.548024 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.548030 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.548036 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.548042 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.548048 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.548054 | controller | 18:54:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.548059 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.548065 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.548071 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.548077 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.548083 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.548089 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.548095 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.548101 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.548106 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.548115 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.548121 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.548127 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.548143 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548166 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548172 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548181 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.548187 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.548193 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.548199 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.548204 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.548213 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.548218 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.548224 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.548230 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.548236 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.548242 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.548248 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.548254 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.548259 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.548265 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.548271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.548277 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.548283 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.548289 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.548294 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.548300 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.548306 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.548312 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.548318 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.548324 | controller | 18:54:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.548329 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.548335 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.548341 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.548347 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.548353 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.548359 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.548365 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.548371 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.548377 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.548382 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.548388 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.548394 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.548400 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548409 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548425 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548431 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.548437 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.548443 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.548449 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.548455 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.548461 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.548467 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.548473 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.548479 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.548484 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.548493 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.548499 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.548505 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.548511 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.548517 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.548523 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.548529 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.548534 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.548540 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.548546 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.548552 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.548558 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.548564 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.548570 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.548576 | controller | 18:54:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.548581 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.548587 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.548593 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.548599 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.548608 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.548614 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.548620 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.548626 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.548632 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.548638 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.548644 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.548650 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.548655 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548661 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548667 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548682 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.548688 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.548695 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.548701 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.548707 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.548716 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.548722 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.548728 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.548733 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.548739 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.548745 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.548751 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.548757 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.548762 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.548768 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.548777 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.548783 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.548789 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.548795 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.548801 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.548807 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.548813 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.548821 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.548827 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.548833 | controller | 18:54:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.548839 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.548845 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.548851 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.548857 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.548863 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.548872 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.548878 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.548884 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.548890 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.548896 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.548901 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.548907 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.548913 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548919 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548925 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.548931 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.548948 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.548955 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.548961 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.548967 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.548973 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.548979 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.548985 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.548990 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.548996 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.549002 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.549008 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.549014 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.549020 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.549029 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.549034 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.549040 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.549046 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.549052 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.549058 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.549064 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.549070 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.549076 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.549082 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.549087 | controller | 18:54:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.549093 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.549099 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.549105 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.549111 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.549117 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.549123 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.549128 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.549134 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.549140 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.549159 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.549168 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.549174 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.549180 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549186 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549192 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549198 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.549204 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.549210 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.549227 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.549235 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.549247 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.549253 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.549258 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.549264 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.549273 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.549279 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.549285 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.549291 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.549297 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.549303 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.549309 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.549314 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.549320 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.549327 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.549332 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.549338 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.549344 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.549349 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.549355 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.549361 | controller | 18:54:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.549367 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.549373 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.549379 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.549385 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.549393 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.549399 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.549405 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.549411 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.549417 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.549423 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.549429 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.549435 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.549440 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549449 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549455 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549461 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.549466 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.549472 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.549478 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.549495 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.549502 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.549508 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.549514 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.549520 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.549525 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.549531 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.549535 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.549540 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.549544 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.549548 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.549552 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.549557 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.549561 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.549565 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.549569 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.549573 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.549578 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.549582 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.549586 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.549590 | controller | 18:54:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.549595 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.549599 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.549603 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.549610 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.549616 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.549620 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.549625 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.549629 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.549633 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.549637 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.549641 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.549646 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.549650 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549654 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549658 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549662 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.549667 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.549671 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.549675 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.549679 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.549691 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.549696 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.549700 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.549704 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.549708 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.549713 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.549719 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.549725 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.549731 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.549737 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.549742 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.549748 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.549754 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.549760 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.549766 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.549772 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.549777 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.549783 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.549788 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.549792 | controller | 18:54:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.549796 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.549800 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.549805 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.549809 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.549813 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.549817 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.549821 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.549826 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.549830 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.549836 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.549842 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.549847 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.549853 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549859 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549865 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.549873 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.549879 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.549885 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.549891 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.549897 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.549905 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.549927 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.549935 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.549944 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.549950 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.549956 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.549962 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.549968 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.549974 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.549984 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.549990 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.549996 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.550002 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.550008 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.550014 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.550020 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.550025 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.550031 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.550037 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.550043 | controller | 18:54:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.550049 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.550054 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.550059 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.550064 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.550069 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.550075 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.550081 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.550087 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.550092 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.550098 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.550104 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.550110 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.550116 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550122 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550177 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550193 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.550200 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.550206 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.550212 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.550218 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.550228 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.550234 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.550240 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.550252 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.550258 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.550264 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.550268 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.550272 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.550276 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.550281 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.550285 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.550289 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.550293 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.550299 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.550305 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.550311 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.550317 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.550323 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.550329 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.550334 | controller | 18:54:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.550340 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.550346 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.550356 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.550362 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.550367 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.550373 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.550379 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.550383 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.550388 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.550392 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.550397 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.550403 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.550409 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550418 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550425 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550430 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.550436 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.550442 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.550448 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.550454 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.550460 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.550466 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.550472 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.550478 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.550494 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.550501 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.550507 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.550513 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.550519 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.550524 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.550538 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.550546 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.550552 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.550558 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.550563 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.550569 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.550575 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.550581 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.550587 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.550593 | controller | 18:54:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.550599 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.550605 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.550611 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.550617 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.550628 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.550634 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.550640 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.550646 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.550652 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.550658 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.550663 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.550669 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.550675 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550681 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550687 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550692 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.550696 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.550701 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.550705 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.550709 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.550713 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.550720 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.550724 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.550728 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.550732 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.550738 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.550754 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.550761 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.550768 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.550774 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.550780 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.550786 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.550791 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.550797 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.550803 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.550809 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.550815 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.550824 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.550830 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.550836 | controller | 18:54:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.550841 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.550848 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.550853 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.550860 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.550865 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.550871 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.550877 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.550883 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.550889 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.550895 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.550901 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.550907 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.550912 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550918 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550924 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.550930 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.550936 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.550942 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.550948 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.550954 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.550960 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.550966 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.550972 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.550978 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.550983 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.550989 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.550995 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.551010 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.551017 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.551027 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.551033 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.551039 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.551045 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.551051 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.551057 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.551062 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.551068 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.551074 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.551080 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.551086 | controller | 18:54:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.551092 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.551103 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.551110 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.551117 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.551126 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.551132 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.551138 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.551144 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.551168 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.551174 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.551180 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.551185 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.551192 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551198 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551204 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551210 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.551216 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.551222 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.551227 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.551233 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.551244 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.551250 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.551256 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.551262 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.551268 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.551274 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.551280 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.551286 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.551292 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.551308 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.551315 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.551321 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.551327 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.551333 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.551338 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.551344 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.551350 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.551356 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.551362 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.551368 | controller | 18:54:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.551374 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.551380 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.551387 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.551392 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.551398 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.551404 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.551410 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.551416 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.551422 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.551428 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.551434 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.551440 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.551445 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551454 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551461 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551467 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.551475 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.551481 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.551487 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.551493 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.551499 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.551505 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.551511 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.551516 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.551522 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.551528 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.551534 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.551540 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.551546 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.551552 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.551567 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.551574 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.551580 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.551586 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.551592 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.551598 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.551604 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.551610 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.551615 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.551621 | controller | 18:54:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.551627 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.551633 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.551639 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.551645 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.551653 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.551659 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.551665 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.551671 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.551677 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.551683 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.551689 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.551694 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.551700 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551706 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551712 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551720 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.551726 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.551732 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.551736 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.551741 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.551745 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.551749 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.551754 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.551758 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.551762 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.551766 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.551771 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.551775 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.551779 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.551784 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.551788 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.551799 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.551804 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.551808 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.551813 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.551817 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.551821 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.551827 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.551834 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.551839 | controller | 18:54:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.551845 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.551849 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.551853 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.551858 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.551862 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.551866 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.551890 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.551894 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.551899 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.551903 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.551907 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.551911 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.551916 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551920 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551924 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.551928 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.551933 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.551937 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.551941 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.551946 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.551950 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.551954 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.551958 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.551963 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.551967 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.551971 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.551975 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.551981 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.551987 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.551998 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.552005 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.552011 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552015 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.552028 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552033 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.552037 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.552041 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.552045 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552050 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.552054 | controller | 18:54:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.552058 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552063 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.552067 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.552071 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.552076 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.552080 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.552084 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.552088 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.552093 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.552101 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.552105 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.552110 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.552114 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552118 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552124 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552128 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.552133 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.552137 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.552141 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.552159 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552168 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552172 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.552177 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.552181 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.552185 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.552189 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.552194 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.552198 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.552202 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.552207 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.552213 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.552217 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552222 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.552228 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552243 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.552251 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.552257 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.552263 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552269 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.552275 | controller | 18:54:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.552281 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552287 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.552293 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.552299 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.552304 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.552309 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.552313 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.552317 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.552322 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.552326 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.552330 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.552334 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.552339 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552346 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552350 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552354 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.552359 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.552363 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.552367 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.552372 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552378 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552383 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.552387 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.552391 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.552396 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.552400 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.552404 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.552409 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.552413 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.552417 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.552421 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.552426 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552430 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.552434 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552439 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.552452 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.552457 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.552461 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552467 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.552472 | controller | 18:54:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.552476 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552480 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.552485 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.552489 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.552495 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.552499 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.552504 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.552508 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.552512 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.552516 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.552521 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.552527 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.552533 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552539 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552545 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552551 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.552558 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.552564 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.552569 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.552575 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552581 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552587 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.552591 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.552596 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.552600 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.552604 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.552608 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.552613 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.552617 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.552621 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.552625 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.552630 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552634 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.552639 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552645 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.552650 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.552655 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.552669 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552674 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.552678 | controller | 18:54:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.552683 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552687 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.552691 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.552696 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.552702 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.552707 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.552711 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.552715 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.552720 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.552724 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.552728 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.552733 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.552737 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552742 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552747 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552751 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.552755 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.552760 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.552764 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.552768 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552773 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552777 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.552782 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.552786 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.552791 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.552795 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.552799 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.552803 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.552808 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.552814 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.552818 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.552822 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552827 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.552831 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.552835 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.552840 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.552844 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.552848 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552859 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.552863 | controller | 18:54:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.552867 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.552872 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.552876 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.552880 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.552885 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.552889 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.552893 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.552898 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.552902 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.552906 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.552911 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.552915 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.552919 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552923 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552928 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.552932 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.552936 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.552940 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.552945 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.552949 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552955 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.552959 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.552964 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.552968 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.552972 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.552976 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.552981 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.552985 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.552989 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.552993 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.552998 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.553002 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553006 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.553010 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553015 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.553021 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.553026 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.553030 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553035 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.553045 | controller | 18:54:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.553050 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553054 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.553058 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.553063 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.553067 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.553071 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.553076 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.553080 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.553084 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.553088 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.553093 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.553097 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.553101 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553109 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553114 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553120 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.553124 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.553128 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.553133 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.553137 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.553141 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.553168 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.553196 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.553200 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.553205 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.553209 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.553213 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.553218 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.553222 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.553226 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.553230 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.553235 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553239 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.553243 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553249 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.553253 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.553257 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.553262 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553266 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.553270 | controller | 18:54:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.553282 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553287 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.553291 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.553295 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.553302 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.553307 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.553311 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.553315 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.553319 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.553324 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.553328 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.553332 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.553340 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553344 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553348 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553353 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.553357 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.553361 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.553366 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.553370 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.553376 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.553380 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.553384 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.553389 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.553393 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.553397 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.553402 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.553406 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.553410 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.553414 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.553419 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.553423 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.553432 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553436 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.553440 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.553444 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.553451 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553455 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.553459 | controller | 18:54:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.553464 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553468 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.553479 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.553484 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.553488 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.553492 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.553496 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.553501 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.553505 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.553509 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.553515 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.553521 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.553527 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553533 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553539 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553545 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.553551 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.553557 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.553561 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.553565 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.553569 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.553574 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.553578 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.553585 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.553589 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.553593 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.553597 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.553602 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.553606 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.553613 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.553622 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.553628 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553636 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.553642 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553648 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.553654 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.553660 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.553665 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553670 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.553674 | controller | 18:54:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.553678 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553682 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.553686 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.553699 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.553704 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.553708 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.553715 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.553721 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.553727 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.553732 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.553738 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.553744 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.553750 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553756 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553762 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553767 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.553773 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.553777 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.553781 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.553785 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.553793 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.553797 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.553801 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.553806 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.553812 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.553819 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.553825 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.553831 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.553837 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.553842 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.553848 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.553854 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553860 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.553866 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.553871 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.553877 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.553883 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.553887 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553892 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.553896 | controller | 18:54:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.553901 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.553907 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.553913 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.553919 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.553925 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.553941 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.553947 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.553953 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.553959 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.553965 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.553971 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.553980 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.553986 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.553995 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554001 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554007 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.554012 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.554018 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.554024 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.554030 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.554036 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.554042 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.554048 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.554054 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.554059 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.554065 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.554071 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.554077 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.554083 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.554089 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.554095 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.554102 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.554109 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.554116 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.554123 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.554129 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.554135 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.554141 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.554168 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.554178 | controller | 18:54:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.554184 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.554190 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.554196 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.554202 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.554213 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.554223 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.554238 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.554245 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.554251 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.554257 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.554263 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.554269 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.554275 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554279 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554284 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554288 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.554292 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.554296 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.554301 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.554305 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.554309 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.554313 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.554318 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.554322 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.554326 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.554330 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.554334 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.554339 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.554343 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.554348 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.554357 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.554362 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.554368 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.554374 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.554380 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.554386 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.554392 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.554400 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.554406 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.554412 | controller | 18:54:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.554418 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.554424 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.554430 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.554436 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.554441 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.554447 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.554453 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.554468 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.554475 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.554481 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.554486 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.554492 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.554498 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554504 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554510 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554516 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.554522 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.554528 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.554534 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.554540 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.554549 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.554555 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.554560 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.554566 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.554572 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.554578 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.554584 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.554590 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.554596 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.554604 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.554610 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.554616 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.554622 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.554628 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.554634 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.554639 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.554645 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.554651 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.554657 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.554663 | controller | 18:54:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.554669 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.554675 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.554680 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.554685 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.554691 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.554697 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.554702 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.554708 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.554714 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.554729 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.554739 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.554745 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.554751 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554756 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554762 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.554770 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.554776 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.554782 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.554788 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.554794 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.554803 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.554808 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.554814 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.554820 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.554826 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.554832 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.554838 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.554843 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.554849 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.554855 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.554861 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.554867 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.554873 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.554879 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.554885 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.554890 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.554896 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.554902 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.554908 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.554914 | controller | 18:54:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.554920 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.554926 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.554932 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.554937 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.554943 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.554949 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.554955 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.554961 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.554967 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.554973 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.554988 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.554994 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.555000 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555009 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555015 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555021 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.555027 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.555033 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.555039 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.555045 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555051 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555057 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.555063 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.555069 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.555075 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.555081 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.555087 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.555095 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.555101 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.555107 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.555113 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.555119 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555125 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.555131 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555137 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.555143 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.555167 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.555173 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.555179 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.555185 | controller | 18:54:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.555191 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.555197 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.555203 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.555209 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.555218 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.555224 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.555230 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.555235 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.555241 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.555247 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.555253 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.555259 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.555274 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555280 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555286 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555292 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.555298 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.555304 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.555310 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.555315 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555319 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555323 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.555328 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.555332 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.555336 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.555341 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.555345 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.555349 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.555353 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.555358 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.555364 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.555369 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555373 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.555377 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555382 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.555386 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.555390 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.555396 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.555401 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.555405 | controller | 18:54:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.555409 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.555414 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.555418 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.555422 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.555427 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.555431 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.555435 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.555439 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.555444 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.555449 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.555454 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.555458 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.555462 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555474 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555478 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555483 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.555487 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.555492 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.555496 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.555500 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555504 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555509 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.555513 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.555517 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.555522 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.555526 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.555530 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.555534 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.555539 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.555545 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.555549 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.555553 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555558 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.555562 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555566 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.555571 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.555575 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.555579 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.555583 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.555588 | controller | 18:54:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.555592 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.555596 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.555601 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.555605 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.555609 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.555614 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.555618 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.555622 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.555626 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.555631 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.555635 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.555639 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.555644 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555648 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555652 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555662 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.555667 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.555671 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.555676 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.555680 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555690 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555695 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.555699 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.555703 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.555708 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.555713 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.555718 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.555722 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.555726 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.555731 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.555735 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.555739 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555745 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.555751 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555757 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.555763 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.555769 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.555775 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.555781 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.555787 | controller | 18:54:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.555794 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.555800 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.555806 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.555812 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.555822 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.555828 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.555834 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.555840 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.555846 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.555852 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.555858 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.555862 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.555885 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555895 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555901 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.555906 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.555923 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.555930 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.555936 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.555942 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555946 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.555951 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.555955 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.555959 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.555964 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.555968 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.555972 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.555976 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.555981 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.555985 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.555989 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.555993 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.555998 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.556002 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556006 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.556011 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.556015 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.556019 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556024 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.556028 | controller | 18:54:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.556032 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556037 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.556041 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.556045 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.556052 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.556056 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.556060 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.556064 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.556071 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.556075 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.556080 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.556084 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.556088 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556093 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556097 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556101 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.556105 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.556117 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.556122 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.556126 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.556131 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.556135 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.556139 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.556155 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.556163 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.556167 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.556172 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.556176 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.556180 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.556184 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.556189 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.556193 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556197 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.556202 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556206 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.556210 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.556214 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.556226 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556231 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.556237 | controller | 18:54:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.556243 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556249 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.556256 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.556262 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.556268 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.556274 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.556280 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.556286 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.556292 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.556298 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.556304 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.556309 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.556315 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556321 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556327 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556336 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.556343 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.556349 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.556355 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.556369 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.556375 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.556381 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.556387 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.556393 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.556402 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.556409 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.556415 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.556421 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.556427 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.556436 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.556441 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.556445 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556450 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.556454 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556458 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.556463 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.556467 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.556471 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556476 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.556480 | controller | 18:54:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.556484 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556489 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.556493 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.556497 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.556502 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.556506 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.556510 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.556514 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.556519 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.556526 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.556532 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.556538 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.556544 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556550 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556556 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556562 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.556567 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.556573 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.556579 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.556583 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.556598 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.556602 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.556607 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.556611 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.556617 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.556623 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.556629 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.556635 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.556641 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.556647 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.556653 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.556659 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556665 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.556671 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556676 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.556682 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.556688 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.556694 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556699 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.556703 | controller | 18:54:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.556708 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556712 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.556718 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.556728 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.556734 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.556740 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.556746 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.556752 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.556758 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.556764 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.556770 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.556776 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.556781 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556790 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556796 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.556802 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.556808 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.556814 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.556820 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.556826 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.556834 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.556854 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.556860 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.556866 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.556872 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.556878 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.556884 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.556889 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.556895 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.556901 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.556911 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.556918 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556924 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.556930 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.556935 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.556941 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.556947 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.556953 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556961 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.556967 | controller | 18:54:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.556973 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.556979 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.556985 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.556991 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.557000 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.557006 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.557012 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.557018 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.557024 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.557030 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.557035 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.557041 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.557047 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557053 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557059 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557065 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.557071 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.557076 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.557082 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.557088 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.557094 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.557100 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.557106 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.557125 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.557132 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.557138 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.557144 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.557166 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.557172 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.557178 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.557184 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.557190 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.557196 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.557202 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.557208 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.557214 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.557219 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.557229 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.557235 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.557241 | controller | 18:54:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.557246 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.557252 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.557258 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.557264 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.557270 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.557276 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.557281 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.557285 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.557289 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.557293 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.557298 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.557302 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.557306 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557310 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557314 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557319 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.557323 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.557327 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.557331 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.557335 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.557340 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.557344 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.557348 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.557352 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.557364 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.557374 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.557381 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.557387 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.557393 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.557401 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.557407 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.557413 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.557419 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.557425 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.557431 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.557436 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.557442 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.557448 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.557454 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.557460 | controller | 18:54:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.557466 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.557472 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.557480 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.557486 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.557494 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.557500 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.557505 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.557511 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.557517 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.557523 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.557529 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.557535 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.557541 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557546 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557552 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557558 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.557564 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.557570 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.557576 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.557581 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.557590 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.557596 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.557602 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.557607 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.557613 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.557619 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.557635 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.557641 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.557647 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.557653 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.557659 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.557665 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.557671 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.557677 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.557683 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.557689 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.557694 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.557700 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.557706 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.557712 | controller | 18:54:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.557718 | controller | \"ac-watcher\" not found\n logger.go:42: 18:54:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.557724 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.557730 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.557736 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.557741 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.557747 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.557753 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.557759 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.557765 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.557771 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.557776 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.557782 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.557788 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557797 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557803 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.557809 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.557815 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.557821 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.557827 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.557833 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.557839 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.557847 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.557853 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.557859 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.557865 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.557871 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.557877 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.557892 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.557899 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.557905 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.557911 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.557917 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.557923 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.557929 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.557935 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.557941 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.557946 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.557952 | controller | expected application_credential_id\"]\n logger.go:42: 18:54:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.557958 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.557964 | controller | 18:55:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.557970 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.557976 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.557982 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.557988 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.557996 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.558002 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.558008 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.558014 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.558020 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.558026 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.558032 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.558037 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.558043 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558049 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558055 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558064 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.558070 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.558076 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.558082 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.558088 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.558096 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.558101 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.558107 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.558113 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.558119 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.558125 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.558131 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.558137 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.558142 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.558179 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.558185 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.558192 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.558197 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.558203 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.558209 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.558215 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.558221 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.558232 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.558238 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.558244 | controller | 18:55:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.558250 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.558258 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.558264 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.558270 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.558276 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.558282 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.558288 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.558294 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.558300 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.558306 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.558312 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.558317 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.558323 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558329 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558335 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558341 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.558347 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.558352 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.558358 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.558364 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.558370 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.558376 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.558382 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.558388 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.558393 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.558399 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.558405 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.558411 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.558417 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.558426 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.558441 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.558448 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.558453 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.558459 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.558465 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.558471 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.558477 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.558482 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.558488 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.558494 | controller | 18:55:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.558500 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.558506 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.558512 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.558517 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.558523 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.558529 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.558535 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.558541 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.558547 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.558553 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.558558 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.558564 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.558570 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558576 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558582 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558588 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.558597 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.558603 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.558609 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.558615 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.558624 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.558631 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.558636 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.558642 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.558648 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.558654 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.558660 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.558666 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.558672 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.558678 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.558686 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.558703 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.558709 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.558715 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.558721 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.558727 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.558732 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.558738 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.558744 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.558750 | controller | 18:55:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.558756 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.558762 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.558767 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.558773 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.558779 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.558785 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.558791 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.558797 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.558803 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.558809 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.558815 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.558820 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.558826 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558835 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558841 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.558846 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.558850 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.558854 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.558858 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.558863 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.558867 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.558871 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.558875 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.558880 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.558884 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.558888 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.558892 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.558896 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.558901 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.558905 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.558909 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.558913 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.558918 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.558929 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.558934 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.558938 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.558943 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.558947 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.558952 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.558956 | controller | 18:55:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.558962 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.558967 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.558971 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.558975 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.558981 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.558986 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.558990 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.558996 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.559002 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.559008 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.559014 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.559020 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.559026 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559032 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559038 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559044 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.559049 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.559055 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.559060 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.559066 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.559072 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.559078 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.559084 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.559090 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.559096 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.559101 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.559107 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.559113 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.559118 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.559124 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.559130 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.559136 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.559143 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.559169 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.559185 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.559192 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.559198 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.559208 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.559214 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.559221 | controller | 18:55:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.559227 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.559233 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.559240 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.559246 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.559255 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.559262 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.559268 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.559274 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.559280 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.559286 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.559292 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.559298 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.559305 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559311 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559320 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559327 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.559333 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.559339 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.559345 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.559351 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.559360 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.559365 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.559369 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.559373 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.559378 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.559383 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.559389 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.559395 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.559401 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.559410 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.559416 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.559423 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.559428 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.559434 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.559441 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.559458 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.559465 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.559470 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.559474 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.559479 | controller | 18:55:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.559483 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.559488 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.559493 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.559500 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.559506 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.559512 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.559518 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.559524 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.559530 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.559536 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.559542 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.559548 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.559554 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559558 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559562 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559567 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.559571 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.559575 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.559580 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.559584 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.559593 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.559599 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.559605 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.559612 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.559618 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.559624 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.559630 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.559636 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.559642 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.559648 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.559654 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.559661 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.559667 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.559673 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.559679 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.559685 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.559708 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.559715 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.559721 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.559727 | controller | 18:55:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.559733 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.559739 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.559745 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.559749 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.559755 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.559763 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.559770 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.559779 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.559785 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.559791 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.559797 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.559803 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.559809 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559818 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559824 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.559830 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.559838 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.559844 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.559849 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.559856 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.559880 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.559887 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.559893 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.559899 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.559905 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.559911 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.559917 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.559923 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.559928 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.559934 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.559940 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.559946 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.559952 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.559958 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.559964 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.559970 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.559976 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.559982 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.559997 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.560003 | controller | 18:55:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.560009 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.560015 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.560021 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.560027 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.560036 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.560042 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.560048 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.560054 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.560060 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.560065 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.560071 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.560077 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.560083 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560092 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560099 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560107 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.560113 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.560120 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.560127 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.560133 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.560139 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.560160 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.560170 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.560176 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.560182 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.560188 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.560194 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.560200 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.560205 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.560211 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.560217 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.560223 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.560229 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.560235 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.560241 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.560247 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.560253 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.560262 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.560268 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.560284 | controller | 18:55:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.560291 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.560297 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.560303 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.560308 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.560314 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.560320 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.560326 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.560332 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.560338 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.560344 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.560350 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.560355 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.560361 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560367 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560373 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560379 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.560385 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.560390 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.560396 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.560402 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.560408 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.560414 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.560420 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.560426 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.560432 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.560438 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.560443 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.560449 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.560455 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.560465 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.560473 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.560479 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.560485 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.560491 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.560497 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.560505 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.560511 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.560517 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.560522 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.560528 | controller | 18:55:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.560544 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.560550 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.560556 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.560562 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.560568 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.560574 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.560580 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.560586 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.560592 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.560597 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.560603 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.560609 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.560615 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560621 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560627 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560633 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.560638 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.560644 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.560650 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.560656 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.560668 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.560674 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.560680 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.560686 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.560692 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.560698 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.560703 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.560709 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.560715 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.560721 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.560727 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.560733 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.560739 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.560745 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.560751 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.560756 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.560762 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.560768 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.560774 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.560780 | controller | 18:55:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.560786 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.560792 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.560807 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.560813 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.560819 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.560825 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.560831 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.560837 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.560843 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.560849 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.560855 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.560861 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.560869 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560878 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560884 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.560890 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.560896 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.560902 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.560908 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.560914 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.560920 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.560926 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.560932 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.560938 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.560943 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.560949 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.560955 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.560961 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.560967 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.560973 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.560978 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.560984 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.560990 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.560996 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561002 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.561008 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.561014 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.561020 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561026 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.561032 | controller | 18:55:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.561038 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561043 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.561049 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.561064 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.561078 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.561084 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.561090 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.561096 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.561102 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.561108 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.561114 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.561119 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.561125 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561131 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561137 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561143 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.561165 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.561172 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.561178 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.561184 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.561189 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.561195 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.561201 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.561207 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.561212 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.561218 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.561224 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.561230 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.561235 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.561241 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.561247 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.561253 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561262 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.561268 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561274 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.561280 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.561286 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.561294 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561300 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.561306 | controller | 18:55:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.561312 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561318 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.561323 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.561329 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.561335 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.561351 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.561357 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.561363 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.561369 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.561375 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.561381 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.561387 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.561393 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561398 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561404 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561410 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.561416 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.561422 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.561426 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.561430 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.561435 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.561439 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.561443 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.561447 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.561452 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.561456 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.561461 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.561467 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.561473 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.561482 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.561488 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.561494 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561500 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.561505 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561511 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.561517 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.561523 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.561529 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561535 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.561541 | controller | 18:55:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.561546 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561552 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.561558 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.561564 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.561570 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.561576 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.561591 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.561597 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.561603 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.561609 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.561614 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.561636 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.561644 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561650 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561656 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561665 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.561671 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.561676 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.561680 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.561684 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.561690 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.561695 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.561699 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.561703 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.561707 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.561712 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.561716 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.561720 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.561724 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.561728 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.561733 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.561737 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561741 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.561745 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561750 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.561754 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.561758 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.561762 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561767 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.561771 | controller | 18:55:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.561775 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561779 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.561784 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.561788 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.561792 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.561796 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.561801 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.561813 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.561817 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.561821 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.561826 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.561830 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.561834 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561840 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561844 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.561849 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.561853 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.561857 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.561861 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.561866 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.561872 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.561876 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.561880 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.561884 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.561889 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.561893 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.561897 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.561901 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.561907 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.561911 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.561916 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.561920 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561924 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.561929 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.561933 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.561937 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.561941 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.561945 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561950 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.561954 | controller | 18:55:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.561958 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.561962 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.561967 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.561971 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.561977 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.561981 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.561985 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.561990 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.561994 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.562004 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.562009 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.562013 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.562017 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562022 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562026 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562030 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.562034 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.562039 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.562043 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.562047 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.562051 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.562056 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.562060 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.562064 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.562068 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.562073 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.562077 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.562081 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.562085 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.562090 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.562096 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.562100 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.562104 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.562108 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.562113 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.562117 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.562121 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.562127 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.562178 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.562190 | controller | 18:55:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.562196 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.562202 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.562208 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.562213 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.562219 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.562225 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.562232 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.562238 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.562244 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.562250 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.562271 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.562279 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.562285 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562291 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562297 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562303 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.562309 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.562315 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.562321 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.562327 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.562333 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.562339 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.562345 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.562351 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.562355 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.562360 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.562364 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.562368 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.562372 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.562381 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.562386 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.562390 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.562396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.562402 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.562408 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.562414 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.562419 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.562425 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.562431 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.562437 | controller | 18:55:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.562443 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.562449 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.562454 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.562459 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.562463 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.562467 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.562471 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.562476 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.562480 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.562484 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.562490 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.562496 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.562513 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562520 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562526 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562532 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.562538 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.562544 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.562550 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.562556 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.562565 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.562571 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.562577 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.562583 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.562589 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.562595 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.562601 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.562610 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.562615 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.562619 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.562624 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.562629 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.562635 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.562641 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.562647 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.562653 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.562659 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.562665 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.562673 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.562679 | controller | 18:55:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.562685 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.562691 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.562697 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.562703 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.562711 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.562716 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.562722 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.562728 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.562734 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.562740 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.562746 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.562752 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.562758 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562777 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562783 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.562789 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.562795 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.562801 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.562807 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.562813 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.562819 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.562825 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.562831 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.562837 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.562842 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.562848 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.562854 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.562860 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.562866 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.562871 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.562877 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.562883 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.562888 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.562894 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.562901 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.562907 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.562913 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.562918 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.562924 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.562930 | controller | 18:55:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.562936 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.562942 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.562948 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.562954 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.562962 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.562968 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.562974 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.562980 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.562986 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.562995 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.563001 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.563006 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.563012 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563018 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563024 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563040 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.563046 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.563052 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.563058 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.563064 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563072 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563078 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.563084 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.563090 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.563096 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.563102 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.563107 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.563113 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.563118 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.563124 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.563130 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.563136 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.563142 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.563166 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.563172 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.563178 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.563184 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.563193 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.563199 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.563205 | controller | 18:55:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.563211 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.563217 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.563223 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.563229 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.563234 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.563240 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.563246 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.563252 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.563258 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.563264 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.563270 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.563275 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.563281 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563287 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563293 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563301 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.563316 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.563320 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.563325 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.563329 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563333 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563337 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.563342 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.563346 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.563350 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.563356 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.563360 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.563365 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.563369 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.563375 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.563379 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.563383 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.563388 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.563392 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.563396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.563400 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.563405 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.563409 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.563413 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.563417 | controller | 18:55:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.563422 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.563426 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.563430 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.563434 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.563439 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.563443 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.563447 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.563451 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.563456 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.563460 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.563464 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.563468 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.563472 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563477 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563481 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563485 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.563489 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.563499 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.563504 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.563508 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563514 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563519 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.563523 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.563527 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.563531 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.563536 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.563540 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.563544 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.563548 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.563553 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.563557 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.563561 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.563566 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.563570 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.563574 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.563578 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.563583 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.563587 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.563591 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.563596 | controller | 18:55:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.563600 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.563604 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.563608 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.563613 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.563617 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.563621 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.563626 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.563630 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.563636 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.563641 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.563645 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.563649 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.563653 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563661 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563665 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563670 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.563674 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.563678 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.563689 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.563693 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563697 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563702 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.563726 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.563730 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.563745 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.563750 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.563754 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.563758 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.563763 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.563767 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.563776 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.563782 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.563789 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.563795 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.563801 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.563828 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.563840 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.563844 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.563849 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.563853 | controller | 18:55:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.563873 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.563881 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.563885 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.563890 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.563899 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.563906 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.563912 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.563918 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.563924 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.563930 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.563936 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.563941 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.563948 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563952 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563956 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.563961 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.563965 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.563969 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.563974 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.563978 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563990 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.563995 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.563999 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.564003 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.564010 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.564015 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.564019 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.564023 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.564028 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.564032 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.564036 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.564041 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564045 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.564049 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564054 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.564058 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.564062 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.564069 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564074 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.564078 | controller | 18:55:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.564082 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564086 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.564091 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.564095 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.564099 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.564104 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.564108 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.564112 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.564117 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.564121 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.564125 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.564129 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.564134 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564138 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564155 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564163 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.564167 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.564172 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.564176 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.564180 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564186 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564198 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.564203 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.564207 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.564211 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.564216 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.564220 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.564224 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.564229 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.564235 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.564239 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.564244 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564248 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.564252 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564256 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.564261 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.564265 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.564269 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564274 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.564278 | controller | 18:55:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.564282 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564287 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.564291 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.564297 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.564302 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.564307 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.564311 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.564315 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.564320 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.564324 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.564328 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.564332 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.564337 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564341 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564345 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564349 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.564354 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.564358 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.564362 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.564367 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564372 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564377 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.564387 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.564392 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.564396 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.564400 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.564405 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.564409 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.564413 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.564418 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.564422 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.564426 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564431 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.564435 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564439 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.564443 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.564448 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.564452 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564456 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.564460 | controller | 18:55:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.564465 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564469 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.564473 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.564478 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.564482 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.564486 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.564491 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.564495 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.564499 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.564504 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.564508 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.564512 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.564516 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564522 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564527 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564531 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.564535 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.564539 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.564544 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.564548 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564552 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564557 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.564561 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.564567 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.564577 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.564581 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.564585 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.564590 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.564594 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.564598 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.564602 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.564607 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564611 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.564615 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564619 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.564624 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.564628 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.564632 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564636 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.564641 | controller | 18:55:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.564645 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564649 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.564653 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.564657 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.564664 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.564668 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.564672 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.564676 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.564681 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.564685 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.564689 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.564693 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.564698 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564702 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564706 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564712 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.564716 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.564720 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.564725 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.564729 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564733 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564737 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.564741 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.564746 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.564750 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.564759 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.564764 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.564768 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.564772 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.564776 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.564780 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.564785 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564789 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.564793 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.564797 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.564802 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.564806 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.564813 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564818 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.564822 | controller | 18:55:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.564826 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.564830 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.564836 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.564840 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.564845 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.564849 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.564853 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.564857 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.564862 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.564868 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.564874 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.564880 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.564886 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564892 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564898 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.564904 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.564910 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.564916 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.564922 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.564928 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564933 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.564938 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.564942 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.564946 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.564950 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.564955 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.564959 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.564973 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.564980 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.564989 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.564996 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.565002 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565008 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.565014 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565020 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.565024 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.565028 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.565033 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565037 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.565041 | controller | 18:55:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.565045 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565050 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.565054 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.565058 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.565062 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.565069 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.565074 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.565078 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.565082 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.565086 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.565091 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.565095 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.565099 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565103 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565108 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565112 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.565116 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.565120 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.565124 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.565129 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565135 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565141 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.565166 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.565174 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.565178 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.565182 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.565186 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.565193 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.565204 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.565209 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.565213 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.565217 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565223 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.565228 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565232 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.565236 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.565240 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.565245 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565249 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.565253 | controller | 18:55:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.565258 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565262 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.565266 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.565270 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.565275 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.565279 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.565283 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.565287 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.565292 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.565296 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.565300 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.565304 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.565308 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565315 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565319 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565323 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.565328 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.565332 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.565336 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.565340 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565345 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565349 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.565353 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.565357 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.565362 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.565366 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.565370 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.565374 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.565379 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.565383 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.565392 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.565397 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565401 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.565405 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565410 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.565414 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.565418 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.565422 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565427 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.565431 | controller | 18:55:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.565435 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565441 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.565446 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.565450 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.565457 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.565461 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.565466 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.565470 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.565474 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.565478 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.565483 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.565487 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.565491 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565495 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565500 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565504 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.565508 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.565512 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.565517 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.565521 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565525 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565529 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.565534 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.565538 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.565542 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.565546 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.565550 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.565555 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.565559 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.565563 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.565567 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.565578 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565583 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.565587 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565591 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.565596 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.565600 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.565606 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565610 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.565615 | controller | 18:55:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.565619 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565623 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.565627 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.565632 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.565637 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.565642 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.565646 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.565650 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.565654 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.565659 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.565663 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.565667 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.565671 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565675 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565680 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565684 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.565690 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.565694 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.565698 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.565702 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565707 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565711 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.565715 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.565719 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.565723 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.565728 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.565732 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.565736 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.565740 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.565747 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.565752 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.565756 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565765 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.565770 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565774 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.565778 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.565782 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.565787 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565791 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.565795 | controller | 18:55:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.565799 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565803 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.565808 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.565812 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.565816 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.565820 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.565825 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.565829 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.565833 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.565837 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.565841 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.565845 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.565850 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565854 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565858 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.565862 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.565867 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.565871 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.565875 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.565879 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565886 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.565890 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.565895 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.565899 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.565903 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.565907 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.565912 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.565916 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.565920 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.565924 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.565928 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.565933 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565937 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.565946 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.565950 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.565955 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.565959 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.565963 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565967 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.565972 | controller | 18:55:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.565978 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.565982 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.565986 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.565990 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.565995 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.565999 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.566003 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.566007 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.566012 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.566016 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.566020 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.566024 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.566028 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566034 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566039 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566044 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.566049 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.566053 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.566057 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.566063 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.566071 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.566077 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.566083 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.566089 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.566095 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.566101 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.566107 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.566113 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.566117 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.566121 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.566126 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.566130 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.566134 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.566138 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.566143 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.566179 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.566187 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.566194 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.566199 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.566206 | controller | 18:55:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.566212 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.566217 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.566222 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.566226 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.566233 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.566237 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.566242 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.566246 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.566250 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.566254 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.566258 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.566263 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.566269 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566275 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566284 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566291 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.566296 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.566302 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.566308 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.566314 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.566320 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.566325 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.566329 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.566333 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.566337 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.566342 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.566346 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.566350 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.566354 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.566359 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.566365 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.566371 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.566377 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.566383 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.566389 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.566395 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.566411 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.566420 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.566426 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.566432 | controller | 18:55:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.566438 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.566442 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.566446 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.566451 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.566456 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.566463 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.566469 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.566475 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.566481 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.566487 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.566493 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.566498 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.566504 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566510 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566516 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566522 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.566528 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.566534 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.566539 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.566545 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.566551 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.566557 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.566563 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.566569 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.566575 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.566581 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.566587 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.566593 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.566599 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.566607 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.566616 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.566622 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.566628 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.566634 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.566640 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.566646 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.566652 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.566657 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.566675 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.566682 | controller | 18:55:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.566687 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.566693 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.566699 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.566704 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.566710 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.566716 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.566722 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.566728 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.566734 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.566740 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.566746 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.566751 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.566757 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566763 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566769 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.566775 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.566781 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.566786 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.566792 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.566798 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.566807 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.566814 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.566820 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.566826 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.566831 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.566837 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.566843 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.566849 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.566855 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.566861 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.566867 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.566873 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.566879 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.566884 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.566890 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.566896 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.566902 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.566908 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.566914 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.566929 | controller | 18:55:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.566936 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.566942 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.566947 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.566953 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.566959 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.566965 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.566971 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.566977 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.566983 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.566989 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.566995 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.567001 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.567006 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567018 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567024 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567030 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.567036 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.567042 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.567048 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.567053 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567059 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567065 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.567071 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.567077 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.567083 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.567088 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.567095 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.567100 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.567104 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.567108 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.567112 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.567117 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567121 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.567125 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567129 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.567134 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.567138 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.567142 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567163 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.567168 | controller | 18:55:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.567180 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567184 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.567189 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.567193 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.567203 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.567207 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.567212 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.567216 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.567220 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.567224 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.567229 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.567233 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.567237 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567241 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567245 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567250 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.567254 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.567258 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.567262 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.567267 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567273 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567277 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.567281 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.567286 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.567290 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.567294 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.567298 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.567302 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.567307 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.567311 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.567315 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.567319 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567324 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.567328 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567332 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.567338 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.567342 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.567348 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567352 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.567357 | controller | 18:55:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.567361 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567365 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.567376 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.567380 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.567385 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.567389 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.567393 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.567397 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.567402 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.567406 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.567410 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.567414 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.567419 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567423 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567427 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567431 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.567435 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.567440 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.567444 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.567448 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567452 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567457 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.567461 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.567465 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.567469 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.567473 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.567478 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.567482 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.567486 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.567492 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.567496 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.567501 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567505 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.567509 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567513 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.567518 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.567522 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.567526 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567532 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.567536 | controller | 18:55:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.567541 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567545 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.567549 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.567559 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.567564 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.567568 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.567572 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.567576 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.567581 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.567585 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.567589 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.567593 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.567599 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567604 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567608 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567614 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.567618 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.567622 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.567627 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.567631 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567637 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567641 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.567646 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.567650 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.567654 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.567658 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.567663 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.567667 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.567671 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.567675 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.567680 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.567684 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567688 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.567692 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567697 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.567701 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.567705 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.567709 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567714 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.567718 | controller | 18:55:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.567722 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567726 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.567731 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.567735 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.567745 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.567750 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.567754 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.567758 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.567762 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.567767 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.567771 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.567775 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.567779 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567785 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567790 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.567795 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.567800 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.567806 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.567812 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.567818 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567824 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.567830 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.567836 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.567842 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.567846 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.567850 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.567902 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.567906 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.567911 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.567915 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.567919 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.567923 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567930 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.567935 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.567939 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.567943 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.567948 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.567952 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567956 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.567960 | controller | 18:55:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.567965 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.567969 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.567973 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.567977 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.567985 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.567990 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.568002 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.568007 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.568011 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.568015 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.568020 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.568024 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.568028 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568033 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568037 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568041 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.568045 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.568049 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.568054 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.568058 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568062 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568066 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.568071 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.568075 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.568079 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.568083 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.568088 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.568092 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.568096 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.568101 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.568107 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.568111 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568115 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.568120 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568124 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.568128 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.568132 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.568139 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568143 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.568163 | controller | 18:55:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.568168 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568172 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.568176 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.568180 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.568185 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.568189 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.568193 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.568204 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.568209 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.568213 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.568217 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.568225 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.568229 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568234 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568238 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568242 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.568246 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.568250 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.568255 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.568259 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568265 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568269 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.568273 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.568277 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.568282 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.568286 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.568290 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.568294 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.568298 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.568305 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.568310 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.568314 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568318 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.568322 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568327 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.568331 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.568335 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.568339 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568344 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.568348 | controller | 18:55:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.568352 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568356 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.568361 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.568365 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.568369 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.568373 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.568377 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.568382 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.568386 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.568397 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.568401 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.568405 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.568410 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568414 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568418 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568422 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.568427 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.568431 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.568435 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.568439 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568446 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568450 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.568454 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.568459 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.568463 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.568467 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.568471 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.568475 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.568481 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.568486 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.568493 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.568499 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568505 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.568511 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568517 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.568522 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.568527 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.568531 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568535 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.568539 | controller | 18:55:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.568544 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568548 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.568552 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.568556 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.568563 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.568567 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.568571 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.568576 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.568580 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.568584 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.568596 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.568601 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.568605 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568611 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568616 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568620 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.568624 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.568629 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.568633 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.568637 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568641 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568646 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.568650 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.568654 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.568658 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.568663 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.568667 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.568671 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.568675 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.568679 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.568684 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.568688 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568692 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.568697 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568701 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.568705 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.568709 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.568714 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568718 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.568722 | controller | 18:55:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.568726 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568731 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.568735 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.568739 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.568745 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.568749 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.568754 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.568758 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.568762 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.568766 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.568772 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.568777 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.568787 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568792 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568796 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568800 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.568804 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.568809 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.568813 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.568817 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568822 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.568826 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.568830 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.568834 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.568838 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.568843 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.568847 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.568851 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.568856 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.568862 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.568868 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.568874 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568879 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.568885 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.568892 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.568897 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.568903 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.568912 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568919 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.568925 | controller | 18:55:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.568931 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.568937 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.568942 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.568946 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.568951 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.568955 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.568959 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.568963 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.568968 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.568972 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.568976 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.568980 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.568984 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.568997 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569002 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569008 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.569013 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.569017 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.569021 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.569026 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569030 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569034 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.569038 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.569042 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.569047 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.569051 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.569055 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.569061 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.569065 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.569072 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.569076 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.569081 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569086 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.569092 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569098 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.569104 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.569110 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.569115 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569119 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.569124 | controller | 18:55:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.569128 | controller | \"ac-watcher\" not found\n logger.go:42: 18:55:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569132 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.569136 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.569141 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.569158 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.569166 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.569170 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.569174 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.569179 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.569183 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.569187 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.569191 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.569195 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569200 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569212 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569216 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.569221 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.569225 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.569229 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.569233 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569242 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569247 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.569251 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.569255 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.569259 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.569264 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.569268 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.569272 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.569276 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.569281 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.569285 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.569289 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569293 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.569298 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569302 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.569306 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.569310 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.569315 | controller | expected application_credential_id\"]\n logger.go:42: 18:55:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569319 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.569323 | controller | 18:55:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.569327 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569332 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.569336 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.569340 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.569344 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.569349 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.569353 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.569357 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.569361 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.569369 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.569373 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.569377 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.569382 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569388 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569392 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569396 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.569406 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.569410 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.569415 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.569419 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569423 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569428 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.569432 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.569436 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.569440 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.569444 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.569449 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.569453 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.569457 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.569461 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.569467 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.569472 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569476 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.569480 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569484 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.569489 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.569493 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.569497 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569501 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.569506 | controller | 18:56:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.569510 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569514 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.569518 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.569522 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.569528 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.569533 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.569537 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.569541 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.569547 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.569553 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.569559 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.569565 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.569571 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569577 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569583 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569589 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.569593 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.569604 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.569608 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.569612 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569617 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569621 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.569625 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.569629 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.569634 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.569640 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.569646 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.569652 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.569658 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.569664 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.569670 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.569676 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569682 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.569688 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569694 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.569699 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.569703 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.569710 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569714 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.569718 | controller | 18:56:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.569723 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569727 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.569731 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.569736 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.569740 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.569749 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.569755 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.569761 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.569767 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.569773 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.569779 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.569785 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.569791 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569797 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569802 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.569807 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.569811 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.569815 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.569826 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.569831 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569835 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.569841 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.569847 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.569853 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.569859 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.569865 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.569871 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.569877 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.569883 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.569895 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.569901 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.569907 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569913 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.569918 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.569922 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.569927 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.569931 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.569937 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569943 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.569949 | controller | 18:56:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.569955 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.569961 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.569967 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.569973 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.569981 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.569987 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.569993 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.569999 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.570007 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.570013 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.570018 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.570024 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.570030 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570036 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570042 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570048 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.570054 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.570060 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.570065 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.570081 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.570090 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.570096 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.570102 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.570108 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.570114 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.570120 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.570126 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.570132 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.570138 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.570163 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.570173 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.570179 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.570185 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.570191 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.570197 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.570203 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.570209 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.570215 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.570221 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.570227 | controller | 18:56:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.570233 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.570239 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.570245 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.570250 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.570256 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.570262 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.570268 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.570274 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.570279 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.570285 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.570291 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.570297 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.570303 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570312 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570318 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570324 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.570330 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.570336 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.570342 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.570348 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.570357 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.570373 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.570379 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.570385 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.570393 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.570399 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.570405 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.570411 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.570417 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.570423 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.570429 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.570433 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.570437 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.570442 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.570446 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.570450 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.570455 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.570459 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.570463 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.570468 | controller | 18:56:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.570472 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.570476 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.570480 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.570485 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.570493 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.570497 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.570502 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.570506 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.570510 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.570514 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.570519 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.570523 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.570527 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570531 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570536 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570542 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.570546 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.570550 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.570555 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.570559 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.570563 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.570568 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.570579 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.570584 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.570588 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.570592 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.570597 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.570601 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.570605 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.570610 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.570614 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.570621 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.570627 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.570633 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.570639 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.570645 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.570651 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.570660 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.570665 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.570669 | controller | 18:56:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.570673 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.570678 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.570682 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.570688 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.570693 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.570697 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.570701 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.570706 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.570710 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.570715 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.570721 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.570727 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.570733 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570739 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570746 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570752 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.570757 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.570764 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.570768 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.570772 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.570777 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.570781 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.570785 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.570790 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.570803 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.570807 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.570812 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.570819 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.570825 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.570834 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.570841 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.570847 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.570852 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.570858 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.570864 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.570870 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.570876 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.570880 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.570884 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.570889 | controller | 18:56:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.570893 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.570897 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.570902 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.570906 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.570912 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.570918 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.570925 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.570931 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.570937 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.570943 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.570949 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.570955 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.570961 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570966 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570972 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.570981 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.570987 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.570991 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.570996 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.571000 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571009 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571015 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.571021 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.571030 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.571037 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.571052 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.571059 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.571065 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.571071 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.571077 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.571085 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.571091 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.571097 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.571103 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.571110 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.571116 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.571122 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.571128 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.571133 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.571139 | controller | 18:56:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.571160 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.571169 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.571175 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.571181 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.571187 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.571193 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.571199 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.571205 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.571211 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.571217 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.571223 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.571229 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.571235 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571244 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571250 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571256 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.571262 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.571268 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.571274 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.571280 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571286 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571292 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.571298 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.571304 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.571310 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.571316 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.571322 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.571338 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.571344 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.571350 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.571356 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.571362 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.571368 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.571374 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.571380 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.571386 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.571392 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.571398 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.571404 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.571410 | controller | 18:56:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.571416 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.571422 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.571436 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.571442 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.571451 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.571457 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.571463 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.571469 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.571475 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.571481 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.571487 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.571493 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.571499 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571505 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571511 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571516 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.571523 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.571529 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.571535 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.571541 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571550 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571556 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.571562 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.571567 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.571573 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.571579 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.571585 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.571591 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.571607 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.571613 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.571619 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.571624 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.571629 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.571633 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.571637 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.571642 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.571646 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.571653 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.571657 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.571662 | controller | 18:56:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.571666 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.571670 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.571675 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.571679 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.571685 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.571689 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.571694 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.571698 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.571702 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.571707 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.571711 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.571715 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.571720 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571724 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571728 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571732 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.571737 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.571741 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.571745 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.571750 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571754 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571761 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.571766 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.571770 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.571774 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.571778 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.571783 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.571787 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.571791 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.571805 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.571810 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.571814 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.571818 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.571823 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.571827 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.571832 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.571836 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.571840 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.571845 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.571867 | controller | 18:56:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.571871 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.571876 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.571880 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.571885 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.571889 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.571893 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.571897 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.571902 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.571907 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.571913 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.571920 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.571926 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.571932 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571938 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571943 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.571947 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.571951 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.571955 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.571960 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.571964 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571971 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.571975 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.571980 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.571984 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.571988 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.571993 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.571997 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.572001 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.572006 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.572010 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.572014 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.572026 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.572031 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.572036 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.572040 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.572044 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.572048 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.572053 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.572057 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.572062 | controller | 18:56:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.572066 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.572074 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.572080 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.572086 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.572092 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.572098 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.572104 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.572110 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.572116 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.572122 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.572128 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.572135 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.572141 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572171 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572178 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572187 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.572193 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.572199 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.572205 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.572211 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.572217 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.572223 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.572229 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.572235 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.572240 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.572246 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.572252 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.572257 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.572263 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.572269 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.572275 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.572281 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.572300 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.572307 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.572313 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.572319 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.572325 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.572332 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.572338 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.572344 | controller | 18:56:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.572351 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.572357 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.572363 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.572370 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.572380 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.572386 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.572392 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.572399 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.572405 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.572411 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.572418 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.572424 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.572430 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572436 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572442 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572448 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.572459 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.572465 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.572471 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.572477 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.572483 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.572489 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.572493 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.572498 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.572502 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.572506 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.572510 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.572515 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.572519 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.572523 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.572527 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.572532 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.572536 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.572553 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.572560 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.572567 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.572572 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.572582 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.572590 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.572596 | controller | 18:56:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.572600 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.572604 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.572608 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.572613 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.572617 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.572621 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.572625 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.572630 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.572634 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.572639 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.572645 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.572651 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.572657 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572663 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572669 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572675 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.572681 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.572686 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.572692 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.572698 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.572705 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.572709 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.572713 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.572718 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.572722 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.572726 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.572730 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.572736 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.572742 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.572751 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.572759 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.572765 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.572771 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.572777 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.572783 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.572799 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.572805 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.572811 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.572817 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.572823 | controller | 18:56:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.572831 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.572837 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.572843 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.572849 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.572855 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.572861 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.572867 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.572873 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.572879 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.572884 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.572890 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.572896 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.572902 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572908 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572914 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.572919 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.572925 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.572931 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.572937 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.572943 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.572952 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.572958 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.572964 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.572970 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.572976 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.572982 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.572988 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.572994 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.572999 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.573005 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.573011 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.573017 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.573023 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.573029 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.573035 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.573040 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.573056 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.573063 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.573069 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.573075 | controller | 18:56:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.573081 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.573087 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.573092 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.573098 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.573104 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.573110 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.573116 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.573122 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.573127 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.573133 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.573138 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.573160 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.573170 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573182 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573191 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573197 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.573203 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.573209 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.573215 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.573221 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.573226 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.573233 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.573238 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.573245 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.573250 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.573256 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.573262 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.573268 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.573274 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.573280 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.573286 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.573292 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.573297 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.573303 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.573309 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.573315 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.573321 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.573342 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.573349 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.573355 | controller | 18:56:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.573361 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.573367 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.573373 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.573379 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.573388 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.573394 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.573399 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.573405 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.573411 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.573417 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.573422 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.573428 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.573434 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573440 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573446 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573451 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.573457 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.573463 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.573469 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.573475 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.573481 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.573487 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.573493 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.573499 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.573504 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.573510 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.573516 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.573523 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.573528 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.573534 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.573540 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.573546 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.573552 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.573558 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.573564 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.573569 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.573575 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.573584 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.573593 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.573609 | controller | 18:56:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.573616 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.573622 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.573628 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.573634 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.573640 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.573646 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.573652 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.573658 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.573663 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.573669 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.573675 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.573681 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.573687 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573693 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573699 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573705 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.573710 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.573716 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.573722 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.573728 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.573734 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.573740 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.573745 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.573751 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.573757 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.573763 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.573769 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.573775 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.573781 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.573790 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.573796 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.573802 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.573808 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.573814 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.573819 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.573825 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.573831 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.573837 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.573843 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.573849 | controller | 18:56:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.573864 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.573871 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.573877 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.573883 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.573889 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.573894 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.573900 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.573906 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.573912 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.573918 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.573924 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.573930 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.573936 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573945 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573951 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.573959 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.573965 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.573971 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.573977 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.573983 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.573993 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.573999 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.574005 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.574011 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.574017 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.574023 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.574029 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.574035 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.574041 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.574047 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.574053 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.574058 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.574064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.574070 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.574076 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.574082 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.574088 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.574094 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.574099 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.574105 | controller | 18:56:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.574111 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.574127 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.574134 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.574140 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.574160 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.574169 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.574175 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.574184 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.574190 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.574196 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.574202 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.574208 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.574214 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574222 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574228 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574234 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.574240 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.574246 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.574252 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.574258 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.574263 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.574269 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.574275 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.574281 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.574287 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.574293 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.574299 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.574305 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.574311 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.574317 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.574323 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.574329 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.574335 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.574340 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.574346 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.574355 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.574361 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.574367 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.574373 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.574379 | controller | 18:56:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.574385 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.574391 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.574397 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.574413 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.574422 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.574428 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.574434 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.574440 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.574446 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.574452 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.574458 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.574464 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.574469 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574475 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574481 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574487 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.574493 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.574499 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.574505 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.574510 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.574516 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.574522 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.574528 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.574534 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.574540 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.574545 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.574551 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.574557 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.574563 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.574569 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.574580 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.574587 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.574592 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.574598 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.574604 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.574610 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.574616 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.574625 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.574631 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.574637 | controller | 18:56:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.574643 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.574649 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.574655 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.574661 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.574677 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.574684 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.574690 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.574695 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.574701 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.574706 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.574710 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.574714 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.574721 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574725 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574730 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574734 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.574738 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.574742 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.574747 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.574751 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.574755 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.574759 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.574764 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.574768 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.574772 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.574776 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.574781 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.574785 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.574789 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.574795 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.574800 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.574804 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.574808 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.574812 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.574817 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.574821 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.574825 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.574829 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.574834 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.574838 | controller | 18:56:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.574842 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.574846 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.574851 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.574855 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.574859 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.574863 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.574875 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.574879 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.574883 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.574888 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.574892 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.574896 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.574901 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574905 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574911 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.574917 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.574923 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.574929 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.574934 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.574940 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.574949 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.574955 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.574961 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.574965 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.574969 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.574973 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.574978 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.574982 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.574986 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.574990 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.574995 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.574999 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.575006 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.575010 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.575014 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.575018 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.575023 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.575027 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.575031 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.575035 | controller | 18:56:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.575040 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.575044 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.575048 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.575052 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.575058 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.575064 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.575070 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.575085 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.575092 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.575098 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.575104 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.575110 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.575115 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575122 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575128 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575134 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.575140 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.575171 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.575179 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.575185 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.575194 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.575200 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.575206 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.575212 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.575218 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.575222 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.575226 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.575230 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.575235 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.575239 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.575243 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.575247 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.575252 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.575258 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.575264 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.575270 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.575275 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.575281 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.575287 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.575293 | controller | 18:56:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.575299 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.575305 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.575311 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.575317 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.575326 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.575334 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.575338 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.575343 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.575359 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.575367 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.575373 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.575378 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.575385 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575390 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575396 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575402 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.575411 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.575417 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.575422 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.575428 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.575434 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.575440 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.575446 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.575452 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.575458 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.575464 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.575470 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.575476 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.575481 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.575487 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.575493 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.575499 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.575505 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.575511 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.575517 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.575523 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.575528 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.575538 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.575544 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.575550 | controller | 18:56:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.575556 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.575561 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.575567 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.575573 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.575579 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.575585 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.575591 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.575597 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.575602 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.575608 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.575625 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.575633 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.575640 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575646 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575652 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575662 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.575668 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.575675 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.575681 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.575687 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.575693 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.575699 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.575705 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.575711 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.575717 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.575722 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.575728 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.575734 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.575743 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.575754 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.575760 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.575766 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.575772 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.575778 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.575784 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.575790 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.575796 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.575802 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.575808 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.575814 | controller | 18:56:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.575819 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.575825 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.575831 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.575837 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.575859 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.575867 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.575873 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.575880 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.575886 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.575891 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.575897 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.575914 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.575920 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575926 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575932 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.575938 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.575944 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.575950 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.575956 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.575962 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.575971 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.575977 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.575983 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.575989 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.575995 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.576001 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.576006 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.576012 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.576018 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.576024 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.576030 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.576036 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.576042 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.576048 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.576054 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.576060 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.576066 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.576072 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.576078 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.576083 | controller | 18:56:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.576090 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.576095 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.576102 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.576109 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.576115 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.576122 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.576129 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.576135 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.576140 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.576164 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.576175 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.576181 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.576186 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576207 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576214 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576220 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.576226 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.576232 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.576238 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.576244 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.576249 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.576255 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.576261 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.576267 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.576273 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.576279 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.576285 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.576291 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.576297 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.576303 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.576312 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.576318 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.576324 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.576330 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.576335 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.576341 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.576347 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.576353 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.576359 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.576365 | controller | 18:56:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.576371 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.576377 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.576383 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.576389 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.576398 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.576404 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.576410 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.576416 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.576422 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.576428 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.576434 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.576439 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.576445 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576451 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576467 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576474 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.576481 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.576487 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.576493 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.576499 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.576507 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.576513 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.576519 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.576525 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.576531 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.576537 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.576543 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.576551 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.576558 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.576564 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.576570 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.576576 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.576582 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.576588 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.576594 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.576600 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.576605 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.576614 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.576620 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.576626 | controller | 18:56:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.576632 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.576638 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.576644 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.576650 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.576656 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.576662 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.576668 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.576674 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.576680 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.576686 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.576692 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.576698 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.576704 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576709 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576715 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576721 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.576736 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.576744 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.576750 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.576756 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.576762 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.576768 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.576774 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.576780 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.576785 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.576791 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.576797 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.576803 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.576809 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.576818 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.576824 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.576830 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.576836 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.576842 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.576848 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.576854 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.576860 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.576866 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.576872 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.576878 | controller | 18:56:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.576883 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.576889 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.576895 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.576901 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.576910 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.576917 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.576923 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.576929 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.576935 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.576943 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.576949 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.576954 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.576960 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576966 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576972 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.576978 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.576984 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.576999 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.577007 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.577013 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577022 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577028 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.577034 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.577039 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.577045 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.577051 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.577057 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.577063 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.577069 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.577075 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.577080 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.577086 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.577092 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.577098 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.577103 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.577107 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.577112 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.577116 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.577122 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.577128 | controller | 18:56:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.577134 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.577140 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.577164 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.577172 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.577178 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.577184 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.577190 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.577196 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.577202 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.577208 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.577214 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.577220 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.577226 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577235 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577241 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577247 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.577253 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.577259 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.577276 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.577282 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577288 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577294 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.577300 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.577306 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.577312 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.577321 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.577327 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.577333 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.577339 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.577344 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.577350 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.577356 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.577362 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.577368 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.577374 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.577380 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.577386 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.577392 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.577396 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.577400 | controller | 18:56:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.577405 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.577409 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.577413 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.577418 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.577425 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.577430 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.577434 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.577438 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.577442 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.577447 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.577451 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.577455 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.577460 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577464 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577468 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577474 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.577479 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.577483 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.577487 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.577499 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577504 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577508 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.577512 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.577517 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.577521 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.577525 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.577529 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.577534 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.577538 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.577542 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.577547 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.577551 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.577555 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.577560 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.577564 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.577569 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.577576 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.577584 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.577591 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.577597 | controller | 18:56:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.577603 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.577607 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.577612 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.577616 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.577620 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.577624 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.577629 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.577633 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.577640 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.577644 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.577648 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.577653 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.577658 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577664 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577670 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577677 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.577683 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.577689 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.577695 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.577701 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577708 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577721 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.577726 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.577730 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.577735 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.577739 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.577743 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.577748 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.577753 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.577762 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.577769 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.577775 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.577781 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.577787 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.577793 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.577799 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.577805 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.577810 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.577815 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.577819 | controller | 18:56:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.577824 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.577828 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.577832 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.577836 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.577841 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.577846 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.577853 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.577859 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.577865 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.577872 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.577877 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.577883 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.577889 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577895 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577901 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.577907 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.577913 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.577919 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.577923 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.577927 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577935 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.577942 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.577958 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.577966 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.577975 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.577982 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.577987 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.577993 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.577999 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.578005 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.578013 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.578019 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578025 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.578031 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578037 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.578043 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.578049 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.578055 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578061 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.578067 | controller | 18:56:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.578073 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578079 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.578085 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.578091 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.578097 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.578106 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.578112 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.578118 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.578124 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.578130 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.578136 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.578142 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.578166 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578175 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578181 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578187 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.578193 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.578199 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.578205 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.578211 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.578217 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.578223 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.578229 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.578234 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.578251 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.578257 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.578263 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.578269 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.578275 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.578281 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.578287 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.578293 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578299 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.578305 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578311 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.578317 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.578323 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.578328 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578338 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.578344 | controller | 18:56:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.578350 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578356 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.578361 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.578369 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.578378 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.578384 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.578390 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.578396 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.578402 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.578408 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.578414 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.578420 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.578426 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578432 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578437 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578443 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.578449 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.578455 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.578461 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.578467 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.578473 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.578479 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.578485 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.578490 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.578494 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.578507 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.578511 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.578516 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.578520 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.578524 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.578529 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.578533 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578537 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.578541 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578546 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.578550 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.578554 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.578561 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578565 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.578569 | controller | 18:56:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.578574 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578578 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.578582 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.578586 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.578595 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.578599 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.578603 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.578607 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.578612 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.578618 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.578624 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.578630 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.578636 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578641 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578647 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578652 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.578658 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.578664 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.578670 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.578674 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.578678 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.578683 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.578687 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.578693 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.578698 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.578702 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.578706 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.578719 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.578723 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.578730 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.578734 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.578738 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578742 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.578747 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578751 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.578755 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.578759 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.578763 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578768 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.578772 | controller | 18:56:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.578776 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578780 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.578784 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.578788 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.578793 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.578797 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.578801 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.578805 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.578809 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.578814 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.578818 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.578822 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.578826 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578830 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578835 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.578839 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.578843 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.578847 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.578852 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.578856 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.578864 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.578869 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.578873 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.578878 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.578882 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.578886 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.578890 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.578894 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.578905 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.578910 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.578914 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.578918 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578923 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.578927 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.578931 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.578935 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.578940 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.578944 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578948 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.578952 | controller | 18:56:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.578957 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.578961 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.578967 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.578971 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.578975 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.578980 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.578984 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.578988 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.578992 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.578997 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.579001 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.579005 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.579009 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579016 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579020 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579026 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.579030 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.579034 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.579038 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.579043 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579047 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579051 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.579055 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.579060 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.579064 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.579068 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.579072 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.579076 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.579081 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.579092 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.579097 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.579103 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579108 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.579112 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579117 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.579122 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.579128 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.579132 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579136 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.579140 | controller | 18:56:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.579157 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579164 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.579169 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.579173 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.579179 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.579184 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.579188 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.579192 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.579196 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.579200 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.579205 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.579209 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.579213 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579217 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579221 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579226 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.579230 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.579234 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.579238 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.579243 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579247 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579254 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.579258 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.579263 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.579267 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.579271 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.579275 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.579279 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.579284 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.579288 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.579292 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.579303 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579308 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.579312 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579316 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.579321 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.579325 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.579331 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579335 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.579339 | controller | 18:56:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.579344 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579348 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.579352 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.579356 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.579361 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.579365 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.579369 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.579373 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.579378 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.579382 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.579386 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.579390 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.579394 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579399 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579403 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579408 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.579414 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.579420 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.579426 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.579432 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579438 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579444 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.579448 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.579452 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.579456 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.579461 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.579465 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.579469 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.579473 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.579480 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.579486 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.579490 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579502 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.579506 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579511 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.579515 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.579519 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.579524 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579528 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.579532 | controller | 18:56:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.579536 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579542 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.579547 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.579551 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.579555 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.579559 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.579563 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.579568 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.579572 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.579576 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.579580 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.579584 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.579589 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579593 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579597 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579601 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.579605 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.579610 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.579614 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.579618 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579625 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579629 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.579634 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.579638 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.579642 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.579648 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.579654 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.579660 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.579665 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.579671 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.579677 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.579682 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579688 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.579701 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579706 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.579710 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.579714 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.579718 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579723 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.579727 | controller | 18:56:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.579731 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579735 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.579740 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.579744 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.579748 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.579752 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.579757 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.579761 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.579765 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.579769 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.579773 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.579778 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.579782 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579791 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579796 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.579800 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.579806 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.579811 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.579816 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.579821 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579829 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.579835 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.579855 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.579861 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.579867 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.579873 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.579878 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.579882 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.579887 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.579891 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.579895 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.579899 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579904 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.579908 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.579919 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.579923 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.579928 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.579932 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579936 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.579941 | controller | 18:56:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.579947 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.579953 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.579959 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.579965 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.579977 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.579984 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.579989 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.579996 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.580001 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.580006 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.580010 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.580014 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.580018 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580023 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580027 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580031 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.580035 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.580040 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.580044 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.580048 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.580052 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.580057 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.580061 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.580065 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.580069 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.580073 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.580078 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.580082 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.580086 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.580090 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.580095 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.580101 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.580105 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.580109 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.580114 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.580118 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.580129 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.580136 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.580141 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.580161 | controller | 18:56:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.580168 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.580173 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.580179 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.580185 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.580191 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.580196 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.580202 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.580208 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.580214 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.580220 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.580226 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.580230 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.580235 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580239 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580243 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580247 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.580251 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.580256 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.580260 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.580264 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.580269 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.580275 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.580281 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.580287 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.580293 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.580299 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.580305 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.580311 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.580317 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.580326 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.580330 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.580335 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.580339 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.580343 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.580348 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.580352 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.580356 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.580371 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.580377 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.580384 | controller | 18:56:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.580390 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.580396 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.580401 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.580407 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.580413 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.580419 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.580425 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.580431 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.580437 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.580442 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.580447 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.580451 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.580455 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580461 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580470 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580478 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.580485 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.580491 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.580496 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.580502 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.580511 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.580517 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.580523 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.580528 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.580534 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.580540 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.580546 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.580552 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.580558 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.580564 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.580570 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.580576 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.580582 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.580588 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.580594 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.580599 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.580605 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.580611 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.580617 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.580633 | controller | 18:56:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.580639 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.580645 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.580651 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.580657 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.580663 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.580672 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.580678 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.580684 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.580690 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.580696 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.580701 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.580707 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.580713 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580722 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580728 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580734 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.580740 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.580746 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.580752 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.580758 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.580764 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.580769 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.580775 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.580781 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.580787 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.580793 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.580799 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.580805 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.580810 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.580816 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.580822 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.580828 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.580834 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.580840 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.580846 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.580852 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.580858 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.580863 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.580872 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.580878 | controller | 18:56:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.580895 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.580901 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.580908 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.580913 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.580924 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.580930 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.580936 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.580942 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.580947 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.580953 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.580959 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.580965 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.580971 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580976 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580982 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.580988 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.580994 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.581000 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.581006 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.581012 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.581021 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.581027 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.581033 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.581039 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.581045 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.581050 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.581056 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.581062 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.581068 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.581074 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.581082 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.581088 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.581094 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.581100 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.581106 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.581112 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.581118 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.581126 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.581132 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.581138 | controller | 18:56:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.581165 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.581185 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.581192 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.581198 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.581204 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.581210 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.581216 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.581222 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.581228 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.581234 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.581240 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.581246 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.581251 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581260 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581266 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581272 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.581278 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.581284 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.581290 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.581296 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.581302 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.581308 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.581314 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.581319 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.581325 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.581331 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.581337 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.581343 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.581349 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.581357 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.581363 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.581369 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.581375 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.581381 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.581385 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.581390 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.581396 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.581402 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.581408 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.581414 | controller | 18:56:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.581420 | controller | \"ac-watcher\" not found\n logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.581426 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.581432 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.581447 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.581453 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.581459 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.581465 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.581471 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.581477 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.581483 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.581489 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.581495 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.581501 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581507 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581512 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581518 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.581524 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.581530 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.581536 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.581542 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.581551 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.581557 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.581563 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.581569 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.581574 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.581580 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.581586 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.581592 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.581598 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.581604 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.581610 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.581616 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.581622 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.581628 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.581634 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.581642 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.581648 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.581654 | controller | expected application_credential_id\"]\n logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.581660 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.581666 | controller | 18:56:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.581672 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.581678 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.581684 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.581690 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.581708 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.581715 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.581721 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.581727 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.581733 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.581739 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.581745 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.581750 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.581756 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581765 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581771 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.581777 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.581783 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.581789 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.581795 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.581801 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.581807 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.581813 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.581819 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.581825 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.581830 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.581836 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.581842 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.581848 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.581854 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.581860 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.581866 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.581872 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.581878 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.581883 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.581889 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.581895 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.581901 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.581907 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.581913 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.581919 | controller | 18:57:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.581925 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.581930 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.581936 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.581942 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.581951 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.581968 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.581974 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.581980 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.581986 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.581992 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.581998 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.582004 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.582013 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582019 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582025 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582031 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.582037 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.582043 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.582048 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.582054 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.582060 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.582066 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.582072 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.582078 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.582083 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.582089 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.582095 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.582101 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.582107 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.582113 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.582119 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.582125 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.582131 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.582137 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.582143 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.582165 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.582172 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.582180 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.582186 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.582192 | controller | 18:57:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.582199 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.582205 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.582211 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.582216 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.582222 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.582228 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.582234 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.582250 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.582256 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.582262 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.582268 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.582274 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.582280 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582286 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582292 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582301 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.582308 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.582314 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.582319 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.582325 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.582339 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.582345 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.582351 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.582357 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.582363 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.582369 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.582375 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.582381 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.582387 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.582398 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.582404 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.582410 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.582418 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.582424 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.582430 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.582436 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.582442 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.582448 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.582454 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.582460 | controller | 18:57:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.582465 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.582471 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.582477 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.582483 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.582489 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.582495 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.582501 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.582507 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.582523 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.582529 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.582535 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.582541 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.582547 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582553 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582559 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582565 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.582570 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.582576 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.582582 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.582588 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.582597 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.582603 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.582609 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.582615 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.582621 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.582627 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.582633 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.582639 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.582645 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.582651 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.582657 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.582663 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.582668 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.582674 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.582680 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.582686 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.582692 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.582698 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.582704 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.582710 | controller | 18:57:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.582716 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.582722 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.582728 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.582734 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.582740 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.582745 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.582751 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.582757 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.582763 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.582769 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.582784 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.582793 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.582799 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582808 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582814 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.582820 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.582826 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.582832 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.582838 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.582844 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.582850 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.582856 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.582862 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.582868 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.582873 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.582879 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.582885 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.582891 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.582897 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.582903 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.582912 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.582918 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.582923 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.582929 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.582934 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.582940 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.582946 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.582952 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.582958 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.582964 | controller | 18:57:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.582970 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.582976 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.582982 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.582988 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.582997 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.583003 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.583009 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.583015 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.583021 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.583027 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.583032 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.583049 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.583055 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583061 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583067 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583073 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.583078 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.583084 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.583090 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.583096 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583102 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583108 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.583114 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.583119 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.583125 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.583131 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.583137 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.583143 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.583169 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.583173 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.583178 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.583182 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583186 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.583191 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583195 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.583199 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.583203 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.583210 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583214 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.583218 | controller | 18:57:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.583222 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583227 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.583231 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.583235 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.583239 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.583244 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.583248 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.583252 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.583256 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.583261 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.583265 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.583269 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.583273 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583286 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583290 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583294 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.583299 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.583303 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.583307 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.583312 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583316 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583320 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.583324 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.583330 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.583335 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.583339 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.583343 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.583348 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.583352 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.583358 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.583362 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.583366 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583371 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.583375 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583379 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.583384 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.583388 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.583392 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583396 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.583401 | controller | 18:57:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.583405 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583409 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.583413 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.583418 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.583423 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.583428 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.583432 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.583436 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.583441 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.583445 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.583451 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.583455 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.583459 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583464 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583474 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583479 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.583483 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.583487 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.583491 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.583496 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583503 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583508 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.583512 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.583516 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.583521 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.583525 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.583529 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.583533 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.583537 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.583542 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.583546 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.583550 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583554 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.583559 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583563 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.583567 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.583571 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.583575 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583583 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.583587 | controller | 18:57:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.583591 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583596 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.583600 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.583604 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.583608 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.583612 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.583617 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.583621 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.583625 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.583629 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.583633 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.583638 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.583642 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583648 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583652 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583657 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.583668 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.583673 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.583677 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.583681 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583686 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583690 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.583694 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.583699 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.583703 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.583707 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.583711 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.583717 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.583722 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.583726 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.583730 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.583735 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583739 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.583743 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583747 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.583751 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.583756 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.583760 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583764 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.583768 | controller | 18:57:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.583772 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583777 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.583781 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.583785 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.583791 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.583795 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.583800 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.583804 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.583808 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.583812 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.583816 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.583821 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.583825 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583829 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583833 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.583855 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.583860 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.583870 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.583875 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.583879 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583883 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.583887 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.583892 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.583896 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.583900 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.583904 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.583908 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.583913 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.583917 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.583921 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.583925 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.583930 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583934 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.583938 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.583942 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.583946 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.583951 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.583957 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583961 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.583965 | controller | 18:57:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.583970 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.583974 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.583978 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.583982 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.583986 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.583991 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.583995 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.583999 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.584003 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.584010 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.584015 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.584019 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.584023 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584027 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584031 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584036 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.584040 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.584044 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.584055 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.584059 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584063 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584067 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.584072 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.584076 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.584080 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.584084 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.584089 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.584093 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.584097 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.584103 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.584107 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.584111 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584116 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.584120 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584124 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.584128 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.584133 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.584137 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584141 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.584160 | controller | 18:57:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.584164 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584169 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.584173 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.584177 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.584181 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.584185 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.584190 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.584194 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.584198 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.584202 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.584207 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.584211 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.584215 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584219 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584223 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584228 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.584232 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.584236 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.584240 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.584251 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584259 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584263 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.584267 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.584271 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.584276 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.584282 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.584286 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.584290 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.584295 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.584299 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.584305 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.584309 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584313 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.584317 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584322 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.584326 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.584330 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.584334 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584338 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.584343 | controller | 18:57:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.584347 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584351 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.584355 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.584360 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.584364 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.584368 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.584372 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.584376 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.584381 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.584385 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.584389 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.584393 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.584397 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584403 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584408 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584412 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.584416 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.584420 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.584424 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.584429 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584440 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584444 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.584449 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.584453 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.584457 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.584461 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.584466 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.584470 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.584474 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.584478 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.584482 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.584487 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584491 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.584495 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584499 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.584504 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.584508 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.584512 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584516 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.584521 | controller | 18:57:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.584525 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584529 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.584533 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.584537 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.584543 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.584548 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.584552 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.584556 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.584562 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.584566 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.584570 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.584575 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.584579 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584583 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584587 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584592 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.584596 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.584600 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.584604 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.584609 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584613 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584617 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.584628 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.584633 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.584637 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.584641 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.584645 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.584650 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.584654 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.584658 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.584662 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.584667 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584671 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.584675 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584680 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.584684 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.584688 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.584696 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584700 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.584704 | controller | 18:57:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.584708 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584713 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.584717 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.584721 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.584727 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.584731 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.584736 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.584740 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.584744 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.584748 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.584752 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.584757 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.584761 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584765 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584769 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584773 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.584778 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.584782 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.584786 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.584790 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584794 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584799 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.584803 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.584812 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.584819 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.584823 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.584827 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.584831 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.584836 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.584842 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.584846 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.584850 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584854 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.584859 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.584863 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.584867 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.584871 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.584875 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584880 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.584884 | controller | 18:57:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.584888 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.584892 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.584897 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.584901 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.584905 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.584909 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.584913 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.584918 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.584922 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.584926 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.584930 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.584935 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.584939 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584943 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584947 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.584951 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.584956 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.584960 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.584964 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.584968 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584975 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.584980 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.584984 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.584988 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.584992 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.585001 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.585006 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.585010 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.585014 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.585019 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.585023 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.585027 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585031 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.585036 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585040 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.585044 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.585048 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.585052 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585056 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.585061 | controller | 18:57:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.585065 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585069 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.585073 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.585079 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.585084 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.585088 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.585092 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.585096 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.585100 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.585105 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.585109 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.585113 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.585117 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585123 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585127 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585134 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.585138 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.585142 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.585160 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.585165 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585169 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585173 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.585178 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.585182 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.585186 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.585190 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.585200 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.585205 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.585209 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.585213 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.585218 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.585222 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585226 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.585230 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585235 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.585239 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.585243 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.585247 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585252 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.585256 | controller | 18:57:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.585260 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585264 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.585269 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.585273 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.585279 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.585283 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.585288 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.585292 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.585296 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.585300 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.585304 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.585309 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.585313 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585317 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585321 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585325 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.585330 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.585334 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.585338 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.585342 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585348 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585353 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.585357 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.585362 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.585367 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.585371 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.585375 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.585379 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.585389 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.585393 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.585397 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.585402 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585406 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.585410 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585415 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.585419 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.585423 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.585429 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585433 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.585438 | controller | 18:57:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.585442 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585446 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.585450 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.585455 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.585459 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.585463 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.585467 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.585472 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.585476 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.585480 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.585484 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.585489 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.585493 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585497 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585501 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585505 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.585510 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.585514 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.585518 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.585522 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585526 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585531 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.585535 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.585539 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.585543 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.585548 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.585552 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.585556 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.585560 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.585571 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.585577 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.585581 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585586 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.585590 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585594 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.585598 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.585603 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.585607 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585611 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.585615 | controller | 18:57:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.585620 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585624 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.585629 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.585634 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.585638 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.585645 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.585649 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.585653 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.585658 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.585662 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.585666 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.585670 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.585675 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585679 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585683 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585687 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.585692 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.585696 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.585700 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.585704 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585710 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585714 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.585719 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.585723 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.585727 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.585731 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.585736 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.585740 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.585744 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.585748 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.585752 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.585762 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585766 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.585771 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585775 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.585779 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.585784 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.585788 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585792 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.585796 | controller | 18:57:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.585800 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585805 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.585809 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.585813 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.585817 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.585821 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.585826 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.585830 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.585834 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.585838 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.585843 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.585847 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.585851 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585857 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585861 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.585866 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.585870 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.585874 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.585878 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.585882 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585887 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.585893 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.585897 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.585901 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.585905 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.585910 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.585914 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.585918 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.585922 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.585927 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.585931 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.585935 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585945 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.585949 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.585954 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.585958 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.585962 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.585966 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585971 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.585975 | controller | 18:57:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.585979 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.585983 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.585988 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.585992 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.586000 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.586004 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.586008 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.586013 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.586017 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.586021 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.586025 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.586030 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.586034 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586038 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586042 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586047 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.586051 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.586055 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.586059 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.586063 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586068 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586072 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.586076 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.586080 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.586085 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.586089 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.586093 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.586097 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.586101 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.586107 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.586112 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.586116 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586120 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.586130 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586134 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.586138 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.586143 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.586163 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586168 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.586172 | controller | 18:57:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.586176 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586182 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.586186 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.586191 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.586195 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.586199 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.586203 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.586208 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.586212 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.586216 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.586220 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.586224 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.586229 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586233 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586237 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586241 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.586245 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.586250 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.586254 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.586258 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586265 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586271 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.586276 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.586282 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.586288 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.586294 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.586300 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.586306 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.586312 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.586321 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.586325 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.586329 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586334 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.586338 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586349 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.586354 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.586358 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.586362 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586366 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.586371 | controller | 18:57:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.586375 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586379 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.586383 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.586388 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.586392 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.586396 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.586400 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.586404 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.586409 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.586413 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.586417 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.586421 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.586426 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586430 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586434 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586441 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.586446 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.586451 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.586455 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.586459 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586465 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586469 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.586474 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.586478 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.586482 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.586486 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.586490 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.586495 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.586499 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.586503 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.586507 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.586512 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586516 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.586520 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586524 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.586529 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.586538 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.586542 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586547 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.586551 | controller | 18:57:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.586555 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586559 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.586564 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.586568 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.586572 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.586576 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.586581 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.586585 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.586589 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.586593 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.586598 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.586602 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.586606 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586612 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586616 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586621 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.586625 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.586629 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.586633 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.586638 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586642 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586646 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.586650 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.586655 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.586659 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.586663 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.586667 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.586672 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.586676 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.586680 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.586684 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.586688 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586693 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.586697 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586701 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.586706 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.586710 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.586719 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586723 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.586727 | controller | 18:57:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.586735 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586739 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.586743 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.586748 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.586754 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.586758 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.586762 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.586767 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.586771 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.586775 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.586779 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.586783 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.586788 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586792 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586796 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586800 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.586804 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.586809 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.586813 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.586817 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586821 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.586826 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.586830 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.586834 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.586838 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.586843 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.586847 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.586851 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.586855 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.586859 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.586865 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.586870 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586874 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.586878 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.586882 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.586886 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.586891 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.586897 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586901 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.586910 | controller | 18:57:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.586914 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.586919 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.586923 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.586927 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.586932 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.586936 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.586940 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.586944 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.586949 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.586953 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.586957 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.586961 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.586965 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586970 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586976 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.586980 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.586984 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.586988 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.586993 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.586997 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587001 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587005 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.587009 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.587014 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.587018 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.587022 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.587026 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.587031 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.587035 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.587042 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.587046 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.587050 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587055 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.587059 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587063 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.587067 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.587071 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.587075 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587080 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.587084 | controller | 18:57:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.587093 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587098 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.587102 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.587106 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.587111 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.587115 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.587119 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.587123 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.587128 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.587132 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.587138 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.587156 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.587166 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587172 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587178 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587184 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.587189 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.587194 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.587198 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.587202 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587211 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587215 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.587220 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.587224 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.587228 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.587232 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.587236 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.587241 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.587245 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.587249 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.587253 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.587257 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587262 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.587266 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587270 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.587274 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.587279 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.587283 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587288 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.587293 | controller | 18:57:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.587297 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587308 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.587313 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.587317 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.587323 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.587328 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.587332 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.587336 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.587340 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.587344 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.587349 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.587353 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.587357 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587363 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587368 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587372 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.587376 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.587380 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.587384 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.587389 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587393 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587397 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.587401 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.587406 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.587410 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.587414 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.587418 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.587422 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.587427 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.587431 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.587435 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.587439 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587444 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.587448 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587452 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.587456 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.587460 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.587465 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587469 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.587473 | controller | 18:57:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.587477 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587482 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.587486 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.587495 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.587501 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.587506 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.587510 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.587514 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.587518 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.587523 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.587529 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.587536 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.587542 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587550 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587556 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587562 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.587567 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.587573 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.587578 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.587583 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587587 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587591 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.587595 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.587599 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.587604 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.587608 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.587612 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.587616 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.587620 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.587624 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.587629 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.587633 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587637 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.587641 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587646 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.587650 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.587654 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.587660 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587665 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.587669 | controller | 18:57:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.587673 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587678 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.587682 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.587686 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.587697 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.587701 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.587706 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.587710 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.587715 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.587719 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.587723 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.587727 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.587731 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587736 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587740 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587746 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.587751 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.587755 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.587760 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.587764 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587768 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587772 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.587776 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.587781 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.587785 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.587789 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.587793 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.587797 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.587802 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.587809 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.587813 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.587818 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587822 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.587826 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.587843 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.587851 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.587855 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.587859 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587865 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.587869 | controller | 18:57:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.587874 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.587878 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.587882 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.587886 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.587891 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.587900 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.587905 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.587909 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.587913 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.587918 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.587922 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.587926 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.587930 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587935 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587939 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.587943 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.587947 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.587952 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.587956 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.587960 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587966 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.587970 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.587975 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.587979 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.587983 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.587987 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.587992 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.587996 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.588000 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.588004 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.588008 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.588013 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.588021 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588025 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.588030 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.588034 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.588038 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588042 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.588047 | controller | 18:57:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.588051 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588055 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.588060 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.588064 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.588069 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.588073 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.588077 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.588088 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.588092 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.588097 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.588101 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.588105 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.588111 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588117 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588121 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588125 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.588130 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.588134 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.588138 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.588142 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588164 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588169 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.588173 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.588177 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.588182 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.588186 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.588190 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.588194 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.588198 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.588203 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.588208 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.588212 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588217 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.588221 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588225 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.588229 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.588234 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.588238 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588242 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.588246 | controller | 18:57:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.588251 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588255 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.588259 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.588263 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.588269 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.588274 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.588278 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.588282 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.588292 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.588297 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.588301 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.588305 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.588310 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588314 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588318 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588322 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.588326 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.588331 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.588335 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.588339 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588343 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588348 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.588352 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.588356 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.588360 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.588364 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.588369 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.588373 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.588377 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.588381 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.588386 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.588390 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.588400 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588404 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.588408 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.588413 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.588419 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588423 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.588427 | controller | 18:57:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.588431 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588435 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.588440 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.588444 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.588448 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.588452 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.588457 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.588461 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.588465 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.588469 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.588479 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.588483 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.588487 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588491 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588496 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588500 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.588504 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.588508 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.588513 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.588517 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588521 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588525 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.588530 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.588534 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.588538 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.588542 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.588546 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.588551 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.588555 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.588561 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.588565 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.588570 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588574 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.588578 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588582 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.588587 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.588591 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.588595 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588599 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.588604 | controller | 18:57:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.588608 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588612 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.588616 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.588620 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.588626 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.588631 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.588635 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.588639 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.588643 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.588647 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.588652 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.588663 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.588667 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588672 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588676 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588680 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.588684 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.588689 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.588693 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.588697 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588705 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588709 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.588713 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.588717 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.588722 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.588726 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.588730 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.588734 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.588739 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.588743 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.588747 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.588751 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588755 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.588760 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588764 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.588768 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.588772 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.588777 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588781 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.588785 | controller | 18:57:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.588790 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588794 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.588798 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.588802 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.588806 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.588811 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.588815 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.588819 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.588823 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.588828 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.588832 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.588836 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.588845 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588852 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588856 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.588860 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.588864 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.588869 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.588873 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.588877 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588881 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.588886 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.588890 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.588894 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.588898 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.588902 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.588907 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.588911 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.588917 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.588921 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.588925 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.588930 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588934 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.588938 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.588942 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.588947 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.588951 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.588955 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588959 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.588964 | controller | 18:57:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.588968 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.588972 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.588976 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.588981 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.588986 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.588991 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.588995 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.588999 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.589003 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.589008 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.589012 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.589016 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.589020 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589024 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589033 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589040 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.589044 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.589048 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.589053 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.589057 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589063 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589067 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.589071 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.589076 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.589080 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.589084 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.589088 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.589092 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.589097 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.589101 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.589105 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.589109 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589114 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.589118 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589122 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.589126 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.589131 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.589137 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589141 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.589156 | controller | 18:57:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.589162 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589166 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.589171 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.589175 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.589179 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.589183 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.589188 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.589192 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.589196 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.589200 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.589206 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.589211 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.589215 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589219 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589223 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589233 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.589238 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.589242 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.589246 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.589251 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589255 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589259 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.589263 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.589268 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.589272 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.589276 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.589280 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.589285 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.589289 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.589295 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.589299 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.589304 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589308 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.589312 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589316 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.589320 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.589325 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.589329 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589333 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.589337 | controller | 18:57:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.589342 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589346 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.589350 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.589354 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.589359 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.589365 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.589371 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.589377 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.589383 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.589388 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.589394 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.589400 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.589406 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589412 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589417 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589421 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.589426 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.589436 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.589441 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.589445 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589452 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589456 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.589460 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.589464 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.589469 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.589473 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.589477 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.589483 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.589488 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.589492 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.589498 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.589502 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589506 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.589511 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589515 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.589519 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.589523 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.589527 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589532 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.589536 | controller | 18:57:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.589540 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589544 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.589549 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.589553 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.589557 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.589561 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.589566 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.589570 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.589574 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.589578 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.589582 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.589587 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.589591 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589597 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589601 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589605 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.589610 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.589614 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.589623 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.589628 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589632 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589636 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.589641 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.589645 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.589649 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.589653 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.589658 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.589662 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.589666 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.589670 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.589674 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.589679 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589683 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.589687 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589691 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.589696 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.589700 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.589704 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589708 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.589713 | controller | 18:57:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.589717 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589721 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.589725 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.589730 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.589736 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.589741 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.589745 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.589749 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.589753 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.589760 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.589765 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.589769 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.589773 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589778 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589782 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589786 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.589790 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.589795 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.589799 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.589808 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589813 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.589817 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.589821 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.589825 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.589830 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.589834 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.589838 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.589842 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.589846 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.589851 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.589855 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.589859 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589863 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.589868 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.589872 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.589876 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.589880 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.589889 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589895 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.589901 | controller | 18:57:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.589907 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.589913 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.589919 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.589925 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.589933 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.589939 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.589943 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.589948 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.589952 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.589956 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.589960 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.589965 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.589969 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589973 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589977 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.589982 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.589986 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.589992 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.589998 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.590004 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590019 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590028 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.590035 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.590040 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.590044 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.590048 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.590052 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.590058 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.590063 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.590069 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.590073 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.590078 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.590082 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.590088 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.590094 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.590100 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.590106 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.590112 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.590118 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.590124 | controller | 18:57:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.590130 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.590136 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.590141 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.590162 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.590166 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.590171 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.590175 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.590179 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.590185 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.590191 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.590197 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.590203 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.590209 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590215 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590221 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590227 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.590233 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.590239 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.590245 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.590250 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590261 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590266 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.590279 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.590286 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.590292 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.590298 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.590304 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.590310 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.590316 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.590322 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.590328 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.590334 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.590339 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.590345 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.590351 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.590357 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.590363 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.590369 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.590375 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.590381 | controller | 18:57:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.590387 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.590393 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.590399 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.590405 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.590411 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.590417 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.590422 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.590428 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.590437 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.590443 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.590449 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.590455 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.590461 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590470 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590476 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590484 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.590490 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.590496 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.590502 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.590508 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590514 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590520 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.590526 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.590541 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.590548 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.590554 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.590560 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.590566 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.590572 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.590578 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.590583 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.590589 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.590595 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.590601 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.590607 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.590613 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.590619 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.590625 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.590631 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.590639 | controller | 18:57:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.590646 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.590652 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.590658 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.590664 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.590672 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.590678 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.590684 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.590690 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.590696 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.590702 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.590708 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.590713 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.590719 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590725 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590731 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590737 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.590743 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.590749 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.590753 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.590757 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590762 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590766 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.590770 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.590774 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.590780 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.590793 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.590798 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.590802 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.590806 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.590810 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.590815 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.590819 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.590823 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.590827 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.590832 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.590836 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.590840 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.590846 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.590850 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.590855 | controller | 18:57:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.590859 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.590863 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.590867 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.590872 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.590876 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.590880 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.590884 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.590889 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.590893 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.590897 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.590901 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.590906 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.590910 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590914 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590918 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.590923 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.590927 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.590931 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.590935 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.590939 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590944 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.590948 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.590952 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.590956 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.590961 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.590965 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.590975 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.590980 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.590984 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.590990 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.590997 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.591002 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591006 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.591011 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591015 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.591019 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.591023 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.591027 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591032 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.591036 | controller | 18:57:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.591040 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591044 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.591049 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.591055 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.591059 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.591063 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.591068 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.591072 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.591076 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.591080 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.591085 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.591089 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.591093 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591097 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591101 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591106 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.591110 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.591114 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.591118 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.591123 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591130 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591135 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.591139 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.591143 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.591163 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.591167 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.591171 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.591175 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.591187 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.591191 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.591196 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.591200 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591204 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.591209 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591213 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.591217 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.591222 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.591226 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591230 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.591234 | controller | 18:57:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.591239 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591243 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.591247 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.591251 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.591256 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.591260 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.591264 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.591268 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.591273 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.591277 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.591281 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.591285 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.591290 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591296 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591300 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591305 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.591309 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.591313 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.591317 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.591322 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591326 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591330 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.591334 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.591349 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.591354 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.591358 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.591363 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.591367 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.591371 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.591383 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.591387 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.591391 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.591400 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591404 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.591409 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.591413 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.591417 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591422 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.591426 | controller | 18:57:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.591430 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591434 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.591439 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.591443 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.591451 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.591455 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.591459 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.591463 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.591468 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.591472 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.591476 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.591480 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.591485 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591489 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591493 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591497 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.591501 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.591506 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.591510 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.591514 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591518 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591523 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.591527 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.591531 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.591535 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.591539 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.591544 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.591548 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.591552 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.591556 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.591567 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.591572 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591576 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.591581 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591585 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.591589 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.591593 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.591599 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591604 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.591608 | controller | 18:57:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.591612 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591617 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.591623 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.591627 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.591631 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.591635 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.591640 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.591644 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.591649 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.591654 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.591660 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.591666 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.591672 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591678 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591684 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591690 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.591696 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.591701 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.591705 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.591709 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591713 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591718 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.591722 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.591726 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.591730 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.591734 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.591739 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.591743 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.591747 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.591755 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.591761 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.591767 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591783 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.591791 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.591797 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.591802 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.591807 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.591811 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591815 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.591820 | controller | 18:57:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.591824 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.591841 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.591845 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.591851 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.591857 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.591864 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.591870 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.591875 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.591881 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.591887 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.591893 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.591899 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.591905 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591910 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591915 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.591922 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.591926 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.591931 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.591935 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.591939 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591947 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.591955 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.591961 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.591967 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.591973 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.591979 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.591985 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.591991 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.591997 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.592003 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.592008 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.592014 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592020 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.592035 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592040 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.592046 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.592052 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.592058 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592064 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.592070 | controller | 18:57:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.592076 | controller | \"ac-watcher\" not found\n logger.go:42: 18:57:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592082 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.592088 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.592094 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.592101 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.592108 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.592114 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.592121 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.592129 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.592134 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.592140 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.592164 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.592170 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592179 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592183 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592188 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.592192 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.592196 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.592200 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.592205 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592213 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592217 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.592221 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.592226 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.592230 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.592234 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.592239 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.592243 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.592247 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.592251 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.592255 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.592260 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592264 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.592268 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592281 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.592285 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.592290 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.592294 | controller | expected application_credential_id\"]\n logger.go:42: 18:57:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592298 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.592302 | controller | 18:58:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.592307 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592313 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.592317 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.592321 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.592327 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.592332 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.592336 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.592340 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.592344 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.592349 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.592353 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.592357 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.592361 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592367 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592373 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592379 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.592385 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.592391 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.592397 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.592402 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592406 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592411 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.592415 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.592419 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.592423 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.592427 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.592432 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.592436 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.592440 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.592444 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.592451 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.592455 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592460 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.592464 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592468 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.592472 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.592484 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.592492 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592498 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.592502 | controller | 18:58:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.592506 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592511 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.592515 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.592519 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.592523 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.592527 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.592532 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.592536 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.592540 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.592544 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.592549 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.592553 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.592557 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592561 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592565 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592570 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.592575 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.592580 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.592584 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.592588 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592592 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592597 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.592601 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.592605 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.592609 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.592614 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.592618 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.592622 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.592626 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.592633 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.592637 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.592641 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592646 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.592650 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592654 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.592658 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.592663 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.592673 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592678 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.592682 | controller | 18:58:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.592687 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592691 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.592695 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.592699 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.592704 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.592708 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.592712 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.592716 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.592721 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.592725 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.592729 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.592733 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.592738 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592742 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592746 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592751 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.592755 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.592759 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.592763 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.592768 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592774 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592779 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.592783 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.592787 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.592792 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.592796 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.592800 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.592804 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.592808 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.592813 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.592817 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.592821 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592825 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.592830 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.592834 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.592838 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.592842 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.592847 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592859 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.592864 | controller | 18:58:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.592868 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.592873 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.592877 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.592881 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.592887 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.592892 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.592896 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.592900 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.592904 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.592909 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.592913 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.592917 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.592921 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592927 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592932 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.592936 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.592940 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.592944 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.592949 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.592953 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592957 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.592961 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.592966 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.592970 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.592974 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.592978 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.592983 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.592987 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.592991 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.592995 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.593000 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.593004 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593008 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.593012 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.593021 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.593025 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.593029 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.593034 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.593044 | controller | 18:58:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.593048 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.593053 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.593057 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.593061 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.593067 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.593072 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.593076 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.593080 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.593085 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.593089 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.593093 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.593097 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.593102 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593106 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593112 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593116 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.593120 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.593124 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.593129 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.593174 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.593181 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.593185 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.593189 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.593194 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.593198 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.593202 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.593206 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.593211 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.593215 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.593219 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.593223 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.593228 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593232 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.593236 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593240 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.593245 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.593249 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.593257 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.593261 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.593265 | controller | 18:58:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.593269 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.593280 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.593284 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.593289 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.593295 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.593301 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.593307 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.593313 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.593319 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.593325 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.593330 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.593336 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.593341 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593346 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593350 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593357 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.593361 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.593366 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.593370 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.593374 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.593378 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.593382 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.593387 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.593391 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.593397 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.593403 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.593409 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.593415 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.593421 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.593430 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.593436 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.593442 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593447 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.593451 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593455 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.593460 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.593464 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.593468 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.593475 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.593479 | controller | 18:58:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.593483 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.593487 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.593502 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.593509 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.593516 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.593522 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.593528 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.593533 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.593539 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.593545 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.593551 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.593555 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.593559 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593564 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593568 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593572 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.593576 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.593580 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.593585 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.593590 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.593600 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.593606 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.593612 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.593618 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.593624 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.593630 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.593635 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.593641 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.593647 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.593653 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.593659 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.593665 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593670 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.593674 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593678 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.593683 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.593689 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.593695 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.593701 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.593707 | controller | 18:58:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.593713 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.593719 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.593725 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.593731 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.593746 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.593753 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.593758 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.593764 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.593770 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.593776 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.593782 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.593788 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.593794 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593806 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593812 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.593818 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.593824 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.593830 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.593836 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.593842 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.593847 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.593853 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.593859 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.593865 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.593871 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.593877 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.593883 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.593889 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.593895 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.593901 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.593909 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.593915 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.593977 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.593984 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.593989 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.593995 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.594001 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.594007 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.594013 | controller | 18:58:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.594019 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.594025 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.594031 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.594036 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.594046 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.594062 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.594069 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.594075 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.594081 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.594114 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.594121 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.594127 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.594133 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594139 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594172 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594184 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.594190 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.594196 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.594202 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.594208 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.594214 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.594220 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.594226 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.594232 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.594238 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.594244 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.594250 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.594256 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.594261 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.594267 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.594274 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.594279 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.594285 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.594291 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.594297 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.594307 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.594314 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.594323 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.594329 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.594334 | controller | 18:58:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.594340 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.594346 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.594351 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.594357 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.594363 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.594369 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.594376 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.594394 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.594402 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.594408 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.594414 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.594421 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.594427 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594433 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594440 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594445 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.594452 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.594458 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.594464 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.594470 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.594480 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.594487 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.594493 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.594499 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.594505 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.594511 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.594517 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.594522 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.594528 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.594537 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.594543 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.594549 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.594555 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.594562 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.594568 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.594575 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.594580 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.594586 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.594592 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.594598 | controller | 18:58:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.594604 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.594610 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.594616 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.594622 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.594631 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.594637 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.594643 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.594649 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.594666 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.594673 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.594679 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.594685 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.594693 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594700 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594706 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594712 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.594718 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.594724 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.594730 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.594736 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.594747 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.594753 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.594759 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.594765 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.594770 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.594776 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.594782 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.594788 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.594794 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.594800 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.594806 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.594811 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.594818 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.594824 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.594830 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.594835 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.594841 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.594847 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.594853 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.594859 | controller | 18:58:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.594865 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.594883 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.594890 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.594895 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.594901 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.594907 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.594913 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.594919 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.594925 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.594942 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.594948 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.594954 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.594960 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594969 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594975 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.594981 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.594987 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.594992 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.594998 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.595004 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.595010 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.595016 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.595022 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.595028 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.595034 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.595040 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.595045 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.595051 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.595057 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.595063 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.595069 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.595075 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.595083 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.595090 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.595096 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.595101 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.595107 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.595113 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.595119 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.595125 | controller | 18:58:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.595131 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.595137 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.595142 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.595167 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.595177 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.595186 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.595192 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.595198 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.595204 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.595210 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.595216 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.595231 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.595238 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595244 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595249 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595255 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.595264 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.595270 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.595276 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.595282 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.595288 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.595294 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.595300 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.595306 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.595311 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.595317 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.595323 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.595329 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.595335 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.595341 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.595347 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.595353 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.595359 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.595365 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.595370 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.595376 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.595382 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.595391 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.595397 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.595403 | controller | 18:58:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.595409 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.595414 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.595420 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.595426 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.595432 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.595438 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.595444 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.595450 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.595456 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.595461 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.595467 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.595476 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.595491 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595497 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595503 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595509 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.595515 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.595521 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.595527 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.595533 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.595539 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.595544 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.595550 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.595556 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.595562 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.595568 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.595574 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.595580 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.595585 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.595595 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.595601 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.595607 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.595613 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.595619 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.595624 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.595630 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.595636 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.595642 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.595648 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.595654 | controller | 18:58:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.595660 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.595666 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.595671 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.595677 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.595683 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.595689 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.595695 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.595701 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.595707 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.595713 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.595719 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.595724 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.595730 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595736 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595751 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.595757 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.595763 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.595769 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.595775 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.595781 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.595793 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.595799 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.595805 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.595811 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.595817 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.595834 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.595840 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.595846 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.595855 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.595860 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.595868 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.595874 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.595880 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.595886 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.595892 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.595898 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.595904 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.595910 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.595916 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.595922 | controller | 18:58:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.595927 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.595933 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.595939 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.595945 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.595951 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.595957 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.595962 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.595968 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.595974 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.595979 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.595984 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.595988 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.595992 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596000 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596005 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596018 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.596022 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.596027 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.596031 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.596035 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596039 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596044 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.596048 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.596052 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.596056 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.596061 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.596065 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.596069 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.596073 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.596077 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.596082 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.596086 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596090 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.596094 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596099 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.596103 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.596107 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.596111 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596116 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.596120 | controller | 18:58:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.596124 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596128 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.596133 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.596137 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.596157 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.596164 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.596168 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.596173 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.596177 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.596181 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.596188 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.596193 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.596197 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596201 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596206 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596210 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.596215 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.596230 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.596237 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.596244 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596250 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596254 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.596258 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.596263 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.596267 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.596271 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.596275 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.596279 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.596284 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.596288 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.596292 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.596296 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596300 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.596305 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596309 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.596313 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.596317 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.596324 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596328 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.596333 | controller | 18:58:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.596337 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596341 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.596345 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.596350 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.596356 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.596360 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.596365 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.596369 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.596373 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.596377 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.596382 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.596386 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.596390 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596394 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596399 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596403 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.596407 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.596411 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.596423 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.596428 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596432 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596436 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.596440 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.596446 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.596451 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.596455 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.596459 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.596465 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.596469 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.596476 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.596480 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.596485 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596489 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.596493 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596498 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.596502 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.596506 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.596510 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596514 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.596519 | controller | 18:58:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.596523 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596527 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.596531 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.596536 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.596540 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.596544 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.596548 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.596552 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.596557 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.596561 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.596565 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.596569 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.596573 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596578 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596582 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596586 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.596590 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.596594 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.596599 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.596610 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596616 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596620 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.596625 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.596629 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.596634 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.596639 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.596645 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.596651 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.596657 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.596663 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.596669 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.596675 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596681 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.596686 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596690 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.596694 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.596698 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.596702 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596707 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.596711 | controller | 18:58:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.596715 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596719 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.596723 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.596728 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.596732 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.596737 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.596743 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.596749 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.596755 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.596764 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.596770 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.596776 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.596782 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596789 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596794 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.596800 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.596804 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.596808 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.596813 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.596817 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596831 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.596838 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.596844 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.596850 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.596856 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.596862 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.596868 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.596874 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.596880 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.596886 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.596892 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.596897 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596901 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.596905 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.596910 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.596914 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.596918 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.596922 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596927 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.596932 | controller | 18:58:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.596938 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.596944 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.596950 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.596956 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.596965 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.596971 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.596977 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.596982 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.596988 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.596994 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.597000 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.597006 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.597011 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597015 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597020 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597024 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.597030 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.597036 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.597043 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.597049 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.597055 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.597061 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.597076 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.597082 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.597088 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.597097 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.597103 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.597109 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.597115 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.597121 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.597127 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.597133 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.597139 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.597158 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.597168 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.597173 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.597179 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.597190 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.597196 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.597202 | controller | 18:58:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.597208 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.597214 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.597220 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.597226 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.597232 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.597237 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.597243 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.597249 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.597255 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.597261 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.597267 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.597273 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.597278 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597284 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597290 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597296 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.597302 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.597308 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.597314 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.597320 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.597326 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.597332 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.597337 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.597353 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.597360 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.597366 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.597372 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.597378 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.597384 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.597393 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.597402 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.597408 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.597414 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.597420 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.597426 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.597432 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.597438 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.597443 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.597449 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.597455 | controller | 18:58:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.597461 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.597467 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.597473 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.597480 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.597486 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.597493 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.597499 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.597505 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.597514 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.597520 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.597526 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.597532 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.597539 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597545 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597551 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597557 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.597563 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.597569 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.597575 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.597581 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.597591 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.597597 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.597603 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.597609 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.597616 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.597634 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.597641 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.597647 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.597653 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.597660 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.597666 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.597672 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.597679 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.597685 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.597691 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.597698 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.597704 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.597710 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.597716 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.597722 | controller | 18:58:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.597728 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.597734 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.597741 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.597747 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.597753 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.597760 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.597766 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.597772 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.597778 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.597784 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.597790 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.597796 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.597802 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597812 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597818 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.597825 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.597831 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.597837 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.597843 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.597850 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.597856 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.597862 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.597868 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.597874 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.597884 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.597890 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.597908 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.597915 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.597921 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.597927 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.597933 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.597939 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.597945 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.597951 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.597956 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.597962 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.597968 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.597974 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.597983 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.597989 | controller | 18:58:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.597995 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.598001 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.598007 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.598013 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.598024 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.598030 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.598036 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.598041 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.598047 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.598053 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.598059 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.598065 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.598071 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598077 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598083 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598089 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.598095 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.598100 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.598108 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.598114 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.598123 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.598131 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.598137 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.598143 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.598178 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.598184 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.598190 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.598206 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.598213 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.598219 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.598225 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.598231 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.598237 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.598243 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.598249 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.598254 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.598260 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.598269 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.598275 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.598281 | controller | 18:58:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.598287 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.598293 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.598299 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.598308 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.598314 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.598320 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.598326 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.598331 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.598337 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.598343 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.598349 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.598355 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.598361 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598366 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598372 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598378 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.598384 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.598390 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.598396 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.598402 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.598408 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.598413 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.598419 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.598425 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.598431 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.598437 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.598443 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.598449 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.598454 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.598474 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.598481 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.598487 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.598493 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.598499 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.598505 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.598511 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.598517 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.598522 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.598528 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.598534 | controller | 18:58:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.598540 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.598546 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.598552 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.598558 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.598564 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.598570 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.598576 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.598582 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.598587 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.598593 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.598599 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.598605 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.598611 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598617 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598623 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598632 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.598638 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.598644 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.598650 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.598656 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.598667 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.598673 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.598679 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.598687 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.598693 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.598699 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.598705 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.598711 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.598717 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.598723 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.598738 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.598745 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.598751 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.598757 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.598763 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.598769 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.598775 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.598780 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.598786 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.598792 | controller | 18:58:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.598798 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.598804 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.598810 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.598815 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.598821 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.598827 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.598832 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.598837 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.598841 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.598845 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.598849 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.598854 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.598858 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598898 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598903 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.598907 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.598911 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.598916 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.598920 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.598924 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.598928 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.598933 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.598937 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.598941 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.598946 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.598950 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.598954 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.598958 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.598962 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.598967 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.598971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.598982 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.598988 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.598994 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599000 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.599006 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.599012 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.599018 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599024 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.599028 | controller | 18:58:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.599032 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599036 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.599044 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.599048 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.599055 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.599059 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.599063 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.599068 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.599073 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.599079 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.599085 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.599091 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.599096 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599102 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599108 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599113 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.599119 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.599124 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.599130 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.599135 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599141 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599174 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.599179 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.599184 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.599188 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.599192 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.599196 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.599201 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.599205 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.599211 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.599220 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.599227 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599233 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.599249 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599256 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.599262 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.599267 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.599274 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599278 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.599283 | controller | 18:58:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.599287 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599291 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.599295 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.599300 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.599304 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.599308 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.599312 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.599317 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.599321 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.599325 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.599329 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.599334 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.599338 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599342 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599349 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599355 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.599361 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.599367 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.599372 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.599378 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599387 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599394 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.599400 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.599406 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.599412 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.599418 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.599423 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.599428 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.599432 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.599438 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.599443 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.599447 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599451 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.599455 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599468 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.599473 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.599477 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.599481 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599485 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.599490 | controller | 18:58:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.599494 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599498 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.599503 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.599507 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.599511 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.599515 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.599520 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.599524 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.599528 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.599532 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.599537 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.599541 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.599545 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599549 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599553 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599560 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.599564 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.599568 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.599573 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.599577 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599583 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599587 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.599591 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.599596 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.599600 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.599604 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.599608 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.599613 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.599617 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.599621 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.599625 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.599630 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599634 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.599638 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599642 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.599653 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.599657 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.599661 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599666 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.599670 | controller | 18:58:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.599674 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599680 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.599684 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.599689 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.599694 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.599699 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.599703 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.599707 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.599711 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.599716 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.599720 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.599724 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.599728 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599734 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599739 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599743 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.599747 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.599751 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.599756 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.599760 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599764 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599768 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.599773 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.599777 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.599781 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.599785 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.599790 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.599794 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.599798 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.599802 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.599807 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.599811 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599815 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.599842 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.599849 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.599854 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.599860 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.599873 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599879 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.599885 | controller | 18:58:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.599891 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.599896 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.599902 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.599908 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.599918 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.599922 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.599926 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.599930 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.599935 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.599939 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.599943 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.599947 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.599951 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599956 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599960 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.599964 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.599971 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.599975 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.599979 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.599984 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599988 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.599992 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.599996 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.600001 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.600005 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.600009 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.600013 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.600017 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.600022 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.600026 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.600030 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.600034 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600038 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.600043 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600047 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.600051 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.600055 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.600061 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600075 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.600081 | controller | 18:58:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.600087 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600093 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.600099 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.600104 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.600110 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.600115 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.600121 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.600126 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.600132 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.600137 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.600158 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.600168 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.600172 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600176 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600180 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600187 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.600192 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.600196 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.600200 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.600204 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600209 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600213 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.600217 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.600221 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.600226 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.600230 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.600234 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.600238 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.600243 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.600249 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.600254 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.600258 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600262 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.600266 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.600275 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.600279 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.600283 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600288 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.600303 | controller | 18:58:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.600308 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600312 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.600316 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.600321 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.600325 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.600329 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.600333 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.600338 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.600342 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.600346 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.600350 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.600355 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.600359 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600363 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600367 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600371 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.600376 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.600380 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.600384 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.600388 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600396 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600402 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.600407 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.600413 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.600419 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.600425 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.600431 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.600437 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.600441 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.600446 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.600450 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.600454 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600458 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.600463 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600467 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.600471 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.600475 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.600479 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600484 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.600488 | controller | 18:58:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.600492 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600505 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.600509 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.600513 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.600518 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.600524 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.600529 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.600533 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.600537 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.600541 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.600546 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.600550 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.600554 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600560 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600565 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600569 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.600574 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.600579 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.600583 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.600587 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600592 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600596 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.600600 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.600604 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.600608 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.600613 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.600617 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.600621 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.600625 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.600629 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.600634 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.600638 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600644 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.600648 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600652 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.600657 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.600661 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.600665 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600669 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.600674 | controller | 18:58:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.600678 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600682 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.600693 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.600698 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.600704 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.600709 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.600713 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.600717 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.600722 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.600726 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.600730 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.600735 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.600739 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600743 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600747 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600752 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.600756 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.600760 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.600765 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.600769 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600773 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600778 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.600782 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.600786 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.600790 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.600795 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.600799 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.600803 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.600808 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.600812 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.600816 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.600820 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600825 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.600829 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.600833 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.600838 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.600842 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.600848 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600854 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.600860 | controller | 18:58:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.600866 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.600873 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.600879 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.600885 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.600900 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.600907 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.600912 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.600916 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.600921 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.600925 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.600929 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.600934 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.600938 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600942 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600947 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.600951 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.600955 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.600960 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.600966 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.600972 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600978 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.600985 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.600991 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.600997 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.601003 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.601009 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.601014 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.601019 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.601215 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.601226 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.601232 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.601238 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.601244 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.601250 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.601256 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.601261 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.601265 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.601269 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.601274 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.601278 | controller | 18:58:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.601282 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.601286 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.601291 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.601295 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.601303 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.601318 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.601325 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.601331 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.601337 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.601343 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.601349 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.601355 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.601361 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601370 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601374 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601378 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.601383 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.601387 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.601391 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.601395 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.601402 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.601406 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.601410 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.601416 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.601422 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.601428 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.601434 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.601439 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.601445 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.601451 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.601457 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.601463 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.601469 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.601474 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.601478 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.601483 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.601487 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.601491 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.601495 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.601500 | controller | 18:58:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.601504 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.601509 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.601515 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.601521 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.601527 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.601533 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.601549 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.601556 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.601562 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.601568 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.601574 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.601580 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.601586 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601593 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601597 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601601 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.601607 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.601613 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.601619 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.601625 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.601634 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.601640 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.601645 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.601651 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.601656 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.601662 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.601668 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.601674 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.601680 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.601685 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.601691 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.601697 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.601703 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.601709 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.601714 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.601723 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.601729 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.601735 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.601740 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.601747 | controller | 18:58:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.601753 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.601759 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.601765 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.601771 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.601779 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.601785 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.601791 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.601797 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.601813 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.601820 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.601826 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.601832 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.601838 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601844 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601850 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.601860 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.601866 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.601872 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.601877 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.601883 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.601889 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.601895 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.601901 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.601906 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.601912 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.601918 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.601924 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.601930 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.601936 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.601941 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.601947 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.601953 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.601959 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.601965 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.601971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.601977 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.601983 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.601992 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.601998 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.602004 | controller | 18:58:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.602009 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.602015 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.602021 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.602027 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.602033 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.602039 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.602044 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.602050 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.602056 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.602073 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.602080 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.602086 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.602106 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602113 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602119 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602125 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.602131 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.602137 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.602143 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.602166 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.602172 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.602177 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.602182 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.602187 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.602193 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.602199 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.602205 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.602211 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.602217 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.602231 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.602237 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.602243 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.602249 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.602255 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.602260 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.602266 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.602271 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.602277 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.602283 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.602289 | controller | 18:58:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.602295 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.602301 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.602306 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.602312 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.602317 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.602323 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.602329 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.602335 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.602340 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.602346 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.602352 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.602370 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.602376 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602382 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602387 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602393 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.602399 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.602405 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.602411 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.602417 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.602426 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.602432 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.602438 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.602444 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.602450 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.602456 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.602462 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.602468 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.602474 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.602479 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.602488 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.602494 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.602503 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.602509 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.602515 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.602521 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.602526 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.602532 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.602538 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.602544 | controller | 18:58:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.602550 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.602556 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.602561 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.602567 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.602573 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.602579 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.602585 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.602595 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.602601 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.602607 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.602613 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.602619 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.602635 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602644 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602650 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602656 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.602662 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.602668 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.602674 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.602680 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.602686 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.602692 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.602698 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.602703 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.602709 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.602715 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.602721 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.602727 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.602732 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.602738 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.602744 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.602750 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.602756 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.602762 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.602768 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.602774 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.602780 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.602786 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.602791 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.602797 | controller | 18:58:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.602803 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.602809 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.602815 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.602821 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.602829 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.602835 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.602841 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.602847 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.602853 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.602859 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.602865 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.602874 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.602880 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602886 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602902 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.602908 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.602914 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.602920 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.602926 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.602932 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.602941 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.602947 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.602953 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.602959 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.602964 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.602970 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.602976 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.602982 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.602988 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.602994 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.602999 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.603005 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.603011 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.603017 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.603023 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.603029 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.603123 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.603132 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.603138 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.603165 | controller | 18:58:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.603177 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.603183 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.603189 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.603195 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.603204 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.603210 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.603215 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.603221 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.603227 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.603233 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.603239 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.603245 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.603251 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603257 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603263 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603279 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.603285 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.603291 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.603297 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.603303 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.603309 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.603315 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.603321 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.603327 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.603332 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.603338 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.603343 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.603349 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.603358 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.603367 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.603373 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.603378 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.603384 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.603390 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.603395 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.603401 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.603407 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.603412 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.603418 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.603424 | controller | 18:58:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.603430 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.603436 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.603442 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.603448 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.603453 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.603459 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.603465 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.603471 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.603477 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.603483 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.603488 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.603494 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.603500 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603506 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603512 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603517 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.603523 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.603540 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.603547 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.603553 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.603562 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.603568 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.603574 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.603580 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.603586 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.603591 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.603597 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.603603 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.603609 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.603615 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.603621 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.603626 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.603632 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.603638 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.603644 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.603650 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.603656 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.603661 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.603670 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.603676 | controller | 18:58:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.603682 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.603688 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.603694 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.603700 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.603706 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.603711 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.603717 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.603723 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.603728 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.603733 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.603739 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.603743 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.603748 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603754 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603758 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603764 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.603768 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.603772 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.603785 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.603790 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.603795 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.603799 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.603803 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.603808 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.603812 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.603835 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.603840 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.603844 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.603848 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.603853 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.603857 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.603861 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.603865 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.603870 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.603876 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.603882 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.603888 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.603894 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.603899 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.603905 | controller | 18:58:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.603911 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.603917 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.603923 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.603929 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.603937 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.603943 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.603948 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.603954 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.603960 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.603966 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.603971 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.603977 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.603982 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603988 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.603994 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604000 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.604006 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.604011 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.604017 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.604034 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604159 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604171 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.604175 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.604180 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.604184 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.604188 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.604192 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.604202 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.604206 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.604211 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.604215 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.604219 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.604223 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.604227 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.604232 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.604236 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.604240 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.604247 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:55 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.604251 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.604256 | controller | 18:58:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.604260 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.604264 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.604268 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.604273 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.604277 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.604281 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.604285 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.604289 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.604294 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.604298 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.604302 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.604306 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.604311 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604315 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604319 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604323 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.604327 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.604332 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.604338 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.604344 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604365 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604372 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.604376 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.604381 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.604385 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.604389 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.604394 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.604398 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.604402 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.604409 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.604415 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.604419 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.604423 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.604428 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.604432 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.604436 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.604441 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.604445 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.604449 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.604453 | controller | 18:58:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.604458 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.604462 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.604466 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.604470 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.604475 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.604479 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.604483 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.604487 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.604492 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.604498 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.604502 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.604506 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.604510 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604515 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604519 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604523 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.604527 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.604532 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.604536 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.604540 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604546 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604558 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.604563 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.604567 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.604572 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.604576 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.604580 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.604584 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.604589 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.604593 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.604597 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.604601 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.604606 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.604610 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.604614 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.604618 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.604623 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.604627 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.604631 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.604635 | controller | 18:58:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.604640 | controller | \"ac-watcher\" not found\n logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.604644 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.604648 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.604652 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.604657 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.604663 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.604669 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.604675 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.604680 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.604686 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.604692 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.604698 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.604704 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604713 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604719 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604725 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.604731 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.604737 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.604742 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.604748 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604754 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604760 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.604766 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.604783 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.604790 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.604800 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.604806 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.604810 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.604815 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.604819 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.604823 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.604827 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.604832 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.604836 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.604840 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.604845 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.604849 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.604853 | controller | expected application_credential_id\"]\n logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.604858 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.604862 | controller | 18:58:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.604866 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.604870 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.604875 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.604879 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.604887 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.604892 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.604896 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.604900 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.604904 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.604909 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.604913 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.604917 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.604921 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604926 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604930 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.604934 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.604938 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.604943 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.604947 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.604951 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604955 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.604960 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.604964 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.604968 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.604980 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.604985 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.604989 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.604994 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.604998 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.605002 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.605006 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.605011 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605015 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.605019 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605023 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.605028 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.605032 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.605038 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605042 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.605047 | controller | 18:59:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.605051 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605055 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.605059 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.605064 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.605068 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.605072 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.605076 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.605080 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.605086 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.605091 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.605095 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.605099 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.605104 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605108 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605112 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605116 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.605121 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.605125 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.605129 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.605133 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.605138 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.605159 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.605171 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.605177 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.605183 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.605187 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.605200 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.605204 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.605208 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.605215 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.605220 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.605224 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605228 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.605232 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605237 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.605243 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.605249 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.605255 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605261 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.605267 | controller | 18:59:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.605273 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605279 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.605285 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.605291 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.605295 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.605299 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.605303 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.605308 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.605312 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.605316 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.605320 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.605325 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.605329 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605333 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605339 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605349 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.605355 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.605361 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.605367 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.605373 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.605386 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.605392 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.605398 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.605402 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.605408 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.605412 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.605417 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.605430 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.605436 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.605442 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.605449 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.605455 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605461 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.605467 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605473 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.605478 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.605484 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.605490 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605496 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.605502 | controller | 18:59:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.605508 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605513 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.605518 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.605522 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.605526 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.605532 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.605538 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.605544 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.605550 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.605556 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.605561 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.605567 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.605573 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605582 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605588 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605594 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.605600 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.605606 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.605612 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.605618 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.605623 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.605629 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.605635 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.605641 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.605647 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.605653 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.605659 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.605665 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.605671 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.605687 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.605693 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.605699 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605705 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.605710 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605716 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.605722 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.605728 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.605734 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605740 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.605745 | controller | 18:59:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.605751 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605757 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.605763 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.605772 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.605781 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.605788 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.605793 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.605799 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.605805 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.605811 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.605817 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.605822 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.605828 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605834 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605840 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.605846 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.605851 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.605857 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.605863 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.605869 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.605874 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.605880 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.605886 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.605892 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.605898 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.605904 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.605909 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.605915 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.605920 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.605926 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.605948 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.605955 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605961 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.605967 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.605973 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.605978 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.605984 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.605992 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.605998 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.606005 | controller | 18:59:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.606011 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.606017 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.606024 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.606030 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.606036 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.606043 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.606049 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.606055 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.606061 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.606067 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.606073 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.606079 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.606085 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606091 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606097 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606103 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.606109 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.606115 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.606121 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.606127 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.606132 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.606138 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.606160 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.606175 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.606182 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.606187 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.606194 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.606200 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.606206 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.606215 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.606220 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.606238 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.606245 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.606252 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.606257 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.606264 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.606269 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.606275 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.606281 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.606287 | controller | 18:59:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.606293 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.606298 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.606304 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.606310 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.606316 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.606327 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.606333 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.606339 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.606345 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.606351 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.606357 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.606363 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.606370 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606375 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606381 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606387 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.606393 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.606399 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.606404 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.606410 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.606420 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.606430 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.606436 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.606442 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.606448 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.606454 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.606460 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.606466 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.606472 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.606477 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.606483 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.606489 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.606495 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.606513 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.606519 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.606525 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.606531 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.606537 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.606543 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.606549 | controller | 18:59:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.606555 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.606561 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.606570 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.606576 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.606584 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.606590 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.606596 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.606601 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.606607 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.606613 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.606619 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.606625 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.606631 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606641 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606648 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606654 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.606660 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.606665 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.606671 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.606677 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.606686 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.606692 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.606697 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.606703 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.606709 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.606715 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.606721 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.606727 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.606733 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.606739 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.606745 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.606750 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.606756 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.606762 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.606778 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.606785 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.606791 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.606797 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.606803 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.606809 | controller | 18:59:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.606815 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.606821 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.606827 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.606833 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.606842 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.606848 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.606854 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.606859 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.606865 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.606871 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.606877 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.606883 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.606889 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606895 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606901 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.606907 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.606912 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.606918 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.606924 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.606930 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.606936 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.606944 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.606950 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.606956 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.606962 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.606968 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.606974 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.606980 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.606986 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.606992 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.606998 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.607003 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607009 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.607015 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607021 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.607038 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.607044 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.607053 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607059 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.607065 | controller | 18:59:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.607071 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607077 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.607083 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.607089 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.607094 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.607100 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.607106 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.607111 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.607117 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.607123 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.607129 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.607135 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.607141 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607164 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607170 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607180 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.607185 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.607191 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.607197 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.607203 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.607209 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.607215 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.607221 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.607227 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.607233 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.607239 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.607244 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.607250 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.607256 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.607265 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.607271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.607277 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607283 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.607289 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607295 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.607301 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.607307 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.607323 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607329 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.607336 | controller | 18:59:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.607341 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607350 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.607356 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.607362 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.607368 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.607374 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.607380 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.607386 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.607392 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.607398 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.607404 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.607410 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.607416 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607421 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607427 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607433 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.607439 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.607445 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.607451 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.607457 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.607466 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.607472 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.607478 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.607484 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.607489 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.607495 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.607501 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.607507 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.607513 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.607519 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.607525 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.607530 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607536 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.607542 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607548 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.607554 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.607560 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.607566 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607579 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.607584 | controller | 18:59:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.607588 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607592 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.607597 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.607601 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.607605 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.607609 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.607614 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.607618 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.607622 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.607626 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.607631 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.607635 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.607639 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607646 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607650 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607655 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.607661 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.607666 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.607670 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.607674 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.607678 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.607683 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.607687 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.607691 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.607695 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.607700 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.607704 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.607708 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.607712 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.607717 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.607722 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.607727 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607731 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.607735 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607740 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.607744 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.607748 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.607752 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607757 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.607768 | controller | 18:59:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.607773 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607777 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.607782 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.607786 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.607792 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.607797 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.607801 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.607805 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.607809 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.607832 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.607836 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.607841 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.607845 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607849 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607853 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.607858 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.607862 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.607866 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.607870 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.607875 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.607881 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.607885 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.607889 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.607893 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.607898 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.607902 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.607906 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.607911 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.607915 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.607919 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.607923 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.607928 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607932 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.607936 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.607940 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.607945 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.607949 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.607955 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607959 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.607964 | controller | 18:59:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.607971 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.607983 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.607988 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.607992 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.607996 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.608000 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.608004 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.608009 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.608013 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.608017 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.608022 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.608026 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.608030 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608034 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608038 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608043 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.608047 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.608051 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.608055 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.608060 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.608064 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.608068 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.608072 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.608077 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.608081 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.608085 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.608089 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.608093 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.608098 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.608104 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.608128 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.608135 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.608141 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.608171 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.608179 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.608185 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.608191 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.608196 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.608202 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.608208 | controller | 18:59:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.608214 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.608219 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.608236 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.608242 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.608251 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.608257 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.608263 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.608268 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.608274 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.608279 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.608284 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.608290 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.608295 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608301 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608309 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608315 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.608320 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.608325 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.608331 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.608337 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.608346 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.608352 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.608357 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.608363 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.608368 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.608374 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.608380 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.608385 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.608391 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.608396 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.608402 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.608407 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.608413 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.608418 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.608424 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.608430 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.608436 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.608442 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.608448 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.608453 | controller | 18:59:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.608460 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.608465 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.608471 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.608477 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.608495 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.608502 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.608508 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.608514 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.608520 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.608526 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.608531 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.608537 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.608543 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608553 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608560 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608566 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.608572 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.608578 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.608584 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.608589 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.608595 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.608600 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.608604 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.608608 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.608612 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.608617 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.608621 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.608625 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.608629 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.608634 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.608638 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.608642 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.608648 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.608654 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.608660 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.608665 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.608671 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.608677 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.608687 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.608693 | controller | 18:59:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.608699 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.608705 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.608711 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.608717 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.608726 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.608746 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.608753 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.608759 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.608765 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.608771 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.608777 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.608783 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.608789 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608795 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608801 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.608811 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.608817 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.608824 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.608830 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.608836 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.608842 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.608848 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.608853 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.608859 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.608866 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.608872 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.608879 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.608885 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.608890 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.608896 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.608903 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.608912 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.608918 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.608924 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.608930 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.608936 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.608942 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.608951 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.608958 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.608964 | controller | 18:59:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.608970 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.608976 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.608982 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.608988 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.608994 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.609000 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.609016 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.609021 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.609025 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.609029 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.609034 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.609038 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.609043 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609050 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609054 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609058 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.609063 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.609067 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.609071 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.609076 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609081 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609087 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.609091 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.609096 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.609100 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.609104 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.609109 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.609113 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.609117 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.609124 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.609128 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.609132 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.609137 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.609141 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.609162 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.609168 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.609172 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.609176 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.609183 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.609187 | controller | 18:59:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.609213 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.609218 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.609222 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.609227 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.609231 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.609236 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.609240 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.609244 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.609255 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.609260 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.609264 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.609269 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.609273 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609277 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609282 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609286 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.609290 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.609294 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.609299 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.609303 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609311 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609316 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.609320 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.609324 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.609328 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.609333 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.609337 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.609341 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.609346 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.609350 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.609357 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.609362 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.609366 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.609371 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.609375 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.609381 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.609385 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.609389 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.609394 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.609398 | controller | 18:59:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.609402 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.609407 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.609411 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.609415 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.609420 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.609426 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.609432 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.609438 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.609444 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.609460 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.609468 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.609473 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.609477 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609484 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609489 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609493 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.609497 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.609501 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.609506 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.609510 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609516 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609522 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.609528 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.609534 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.609540 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.609546 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.609552 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.609558 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.609564 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.609568 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.609573 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.609577 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.609581 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.609586 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.609590 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.609594 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.609599 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.609603 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.609608 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.609614 | controller | 18:59:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.609620 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.609626 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.609632 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.609638 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.609647 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.609653 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.609659 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.609665 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.609671 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.609677 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.609683 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.609703 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.609711 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609718 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609724 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609730 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.609736 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.609742 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.609748 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.609754 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609760 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609766 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.609772 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.609778 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.609784 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.609790 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.609796 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.609802 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.609808 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.609814 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.609820 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.609826 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.609832 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.609838 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.609844 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.609850 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.609856 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.609866 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.609871 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.609876 | controller | 18:59:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.609880 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.609884 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.609889 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.609893 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.609900 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.609904 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.609908 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.609913 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.609917 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.609921 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.609926 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.609930 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.609945 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609950 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609954 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.609959 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.609963 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.609967 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.609972 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.609976 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609980 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.609986 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.609992 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.609998 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.610004 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.610010 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.610016 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.610022 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.610028 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.610037 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.610043 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.610049 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.610057 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.610064 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.610070 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.610075 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.610079 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.610084 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.610088 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.610092 | controller | 18:59:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.610096 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.610101 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.610105 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.610109 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.610113 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.610118 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.610122 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.610126 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.610130 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.610134 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.610139 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.610143 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.610174 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610192 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610199 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610205 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.610211 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.610217 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.610223 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.610229 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.610240 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.610246 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.610252 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.610259 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.610265 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.610271 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.610276 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.610282 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.610288 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.610294 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.610300 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.610306 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.610312 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.610318 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.610324 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.610329 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.610335 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.610341 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.610347 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.610353 | controller | 18:59:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.610359 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.610365 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.610371 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.610377 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.610383 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.610389 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.610394 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.610400 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.610406 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.610412 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.610418 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.610427 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.610434 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610443 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610449 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610467 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.610474 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.610480 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.610486 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.610491 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.610497 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.610503 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.610509 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.610515 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.610521 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.610527 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.610533 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.610539 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.610545 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.610550 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.610556 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.610562 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.610568 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.610574 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.610580 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.610586 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.610592 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.610598 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.610604 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.610609 | controller | 18:59:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.610616 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.610621 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.610627 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.610633 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.610644 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.610650 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.610656 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.610662 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.610668 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.610673 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.610680 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.610685 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.610691 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610697 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610703 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610709 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.610725 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.610731 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.610737 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.610743 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.610748 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.610754 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.610760 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.610766 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.610772 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.610778 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.610784 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.610790 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.610799 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.610805 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.610811 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.610817 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.610823 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.610829 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.610835 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.610841 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.610847 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.610856 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.610862 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.610868 | controller | 18:59:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.610874 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.610880 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.610886 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.610892 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.610897 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.610903 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.610909 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.610915 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.610921 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.610927 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.610933 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.610939 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.610945 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610951 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610957 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.610963 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.610969 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.610975 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.610991 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.610997 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611003 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611009 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.611015 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.611021 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.611025 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.611029 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.611033 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.611037 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.611042 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.611049 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.611055 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.611060 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.611069 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611075 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.611081 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.611087 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.611093 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611099 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.611106 | controller | 18:59:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.611113 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611120 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.611127 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.611133 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.611139 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.611159 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.611168 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.611174 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.611180 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.611186 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.611195 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.611201 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.611207 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611213 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611218 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611224 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.611230 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.611236 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.611242 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.611258 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611267 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611273 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.611279 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.611285 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.611290 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.611294 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.611298 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.611302 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.611307 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.611311 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.611315 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.611319 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611324 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.611328 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611332 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.611336 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.611341 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.611345 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611349 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.611353 | controller | 18:59:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.611358 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611362 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.611366 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.611370 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.611374 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.611379 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.611383 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.611387 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.611391 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.611396 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.611400 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.611404 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.611408 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611414 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611419 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611423 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.611427 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.611431 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.611436 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.611440 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611453 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611458 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.611462 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.611466 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.611471 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.611475 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.611479 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.611485 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.611489 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.611493 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.611498 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.611502 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611506 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.611510 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611515 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.611519 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.611523 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.611527 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611532 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.611536 | controller | 18:59:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.611540 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611544 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.611549 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.611553 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.611560 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.611565 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.611569 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.611573 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.611577 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.611582 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.611586 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.611590 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.611594 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611598 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611603 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611607 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.611611 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.611615 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.611619 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.611624 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611628 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611639 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.611643 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.611648 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.611652 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.611656 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.611660 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.611665 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.611669 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.611673 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.611677 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.611681 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611686 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.611690 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611694 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.611698 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.611702 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.611708 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611713 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.611717 | controller | 18:59:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.611721 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611726 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.611730 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.611734 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.611738 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.611745 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.611749 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.611753 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.611757 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.611762 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.611766 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.611770 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.611774 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611779 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611783 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611787 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.611793 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.611797 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.611801 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.611805 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611823 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.611828 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.611833 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.611847 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.611852 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.611856 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.611860 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.611865 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.611869 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.611875 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.611879 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.611884 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611888 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.611892 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.611897 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.611901 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.611905 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.611910 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611914 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.611918 | controller | 18:59:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.611922 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.611927 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.611933 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.611939 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.611944 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.611950 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.611956 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.611962 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.611968 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.611974 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.611979 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.611983 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.611987 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611991 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.611996 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612002 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.612007 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.612011 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.612015 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.612019 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612026 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612031 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.612037 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.612043 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.612059 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.612069 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.612075 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.612081 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.612087 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.612091 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.612095 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.612100 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612104 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.612108 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612112 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.612117 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.612121 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.612125 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.612129 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.612135 | controller | 18:59:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.612141 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.612166 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.612172 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.612178 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.612184 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.612190 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.612196 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.612200 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.612204 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.612208 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.612213 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.612217 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.612221 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612228 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612233 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612237 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.612241 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.612246 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.612250 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.612256 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612262 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612268 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.612274 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.612280 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.612286 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.612291 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.612308 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.612315 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.612320 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.612326 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.612332 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.612338 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612342 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.612346 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612351 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.612357 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.612363 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.612369 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.612375 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.612381 | controller | 18:59:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.612387 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.612393 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.612399 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.612405 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.612414 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.612420 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.612425 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.612431 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.612440 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.612446 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.612452 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.612458 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.612464 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612470 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612476 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612482 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.612488 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.612494 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.612500 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.612505 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612514 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612520 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.612525 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.612531 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.612537 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.612543 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.612549 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.612565 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.612571 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.612575 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.612581 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.612586 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612590 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.612594 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612599 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.612603 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.612607 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.612614 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.612618 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.612623 | controller | 18:59:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.612627 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.612631 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.612635 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.612640 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.612644 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.612648 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.612652 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.612657 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.612661 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.612665 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.612670 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.612674 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.612678 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612682 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612687 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612691 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.612695 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.612699 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.612704 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.612708 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612712 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612716 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.612721 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.612725 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.612731 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.612735 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.612740 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.612744 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.612748 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.612761 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.612766 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.612770 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612775 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.612779 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612783 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.612788 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.612792 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.612796 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.612800 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.612805 | controller | 18:59:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.612809 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.612814 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.612819 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.612825 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.612831 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.612837 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.612843 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.612849 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.612855 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.612860 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.612864 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.612869 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.612873 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612877 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612881 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.612885 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.612890 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.612894 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.612898 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.612903 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612909 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.612914 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.612920 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.612926 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.612932 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.612938 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.612944 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.612949 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.612955 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.612961 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.612976 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.612980 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612985 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.612989 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.612993 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.612998 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.613002 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.613006 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613010 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.613015 | controller | 18:59:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.613019 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613023 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.613028 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.613035 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.613047 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.613052 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.613056 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.613060 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.613065 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.613069 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.613073 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.613077 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.613082 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613088 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613092 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613097 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.613101 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.613105 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.613110 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.613114 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613118 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613122 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.613127 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.613131 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.613135 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.613139 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.613156 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.613164 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.613168 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.613172 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.613176 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.613188 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613192 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.613197 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613201 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.613205 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.613209 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.613214 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613218 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.613222 | controller | 18:59:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.613227 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613231 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.613235 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.613239 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.613246 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.613250 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.613254 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.613258 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.613263 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.613267 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.613271 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.613275 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.613280 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613284 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613288 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613292 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.613296 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.613301 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.613305 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.613309 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613313 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613318 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.613322 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.613328 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.613332 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.613337 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.613341 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.613345 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.613349 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.613354 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.613358 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.613362 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613366 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.613377 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613382 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.613386 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.613390 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.613396 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613401 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.613405 | controller | 18:59:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.613409 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613414 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.613418 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.613422 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.613426 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.613431 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.613435 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.613439 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.613443 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.613448 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.613452 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.613456 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.613460 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613465 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613469 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613475 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.613479 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.613484 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.613488 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.613492 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613498 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613502 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.613507 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.613511 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.613515 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.613519 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.613524 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.613528 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.613532 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.613538 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.613542 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.613547 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613551 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.613555 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613566 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.613570 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.613574 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.613579 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613583 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.613587 | controller | 18:59:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.613592 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613596 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.613602 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.613606 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.613610 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.613615 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.613619 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.613623 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.613627 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.613632 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.613636 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.613640 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.613644 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613649 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613653 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613657 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.613661 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.613666 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.613670 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.613674 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613680 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613684 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.613688 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.613693 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.613697 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.613701 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.613705 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.613710 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.613714 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.613718 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.613722 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.613727 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613731 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.613735 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613739 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.613750 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.613755 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.613759 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613763 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.613768 | controller | 18:59:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.613772 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613776 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.613780 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.613785 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.613789 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.613793 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.613797 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.613802 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.613806 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.613810 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.613814 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.613818 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.613823 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613829 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613833 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.613837 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.613842 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.613846 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.613850 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.613854 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613859 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.613865 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.613869 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.613873 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.613877 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.613882 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.613886 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.613890 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.613894 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.613899 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.613904 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.613909 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613913 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.613917 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.613922 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.613926 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.613930 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.613941 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613945 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.613949 | controller | 18:59:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.613954 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.613958 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.613962 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.613967 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.613973 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.613977 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.613981 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.613986 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.613990 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.613994 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.613998 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.614002 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.614007 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614011 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614015 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614019 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.614024 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.614028 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.614032 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.614036 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614040 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614045 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.614049 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.614053 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.614057 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.614062 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.614066 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.614070 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.614076 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.614082 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.614088 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.614094 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614101 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.614106 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614112 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.614119 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.614125 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.614135 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614177 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.614189 | controller | 18:59:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.614195 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614204 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.614210 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.614216 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.614222 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.614228 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.614232 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.614237 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.614241 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.614245 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.614249 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.614254 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.614258 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614262 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614266 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614270 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.614275 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.614279 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.614283 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.614288 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614292 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614296 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.614300 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.614304 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.614309 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.614313 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.614317 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.614321 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.614326 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.614332 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.614337 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.614341 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614345 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.614349 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614354 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.614358 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.614362 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.614366 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614370 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.614383 | controller | 18:59:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.614388 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614392 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.614397 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.614401 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.614407 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.614412 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.614416 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.614420 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.614424 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.614428 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.614433 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.614437 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.614441 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614445 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614449 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614455 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.614461 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.614465 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.614469 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.614474 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614481 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614485 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.614489 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.614494 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.614498 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.614502 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.614506 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.614510 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.614515 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.614519 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.614523 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.614527 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614532 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.614536 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614540 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.614544 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.614549 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.614553 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614557 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.614561 | controller | 18:59:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.614572 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614577 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.614581 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.614585 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.614590 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.614594 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.614598 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.614602 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.614607 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.614611 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.614615 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.614619 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.614624 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614630 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614634 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614638 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.614642 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.614647 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.614651 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.614655 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614659 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614664 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.614668 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.614672 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.614676 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.614681 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.614685 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.614689 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.614693 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.614697 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.614702 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.614706 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614710 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.614714 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614719 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.614723 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.614727 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.614732 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614736 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.614740 | controller | 18:59:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.614746 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614750 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.614762 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.614766 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.614772 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.614777 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.614781 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.614785 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.614790 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.614794 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.614798 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.614802 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.614807 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614811 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614815 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614821 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.614825 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.614829 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.614834 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.614838 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614842 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.614846 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.614851 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.614855 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.614859 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.614863 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.614868 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.614872 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.614876 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.614881 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.614885 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.614889 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614893 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.614898 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.614902 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.614906 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.614910 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.614916 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614920 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.614925 | controller | 18:59:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.614929 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.614933 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.614937 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.614948 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.614953 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.614957 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.614961 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.614965 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.614970 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.614974 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.614978 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.614982 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.614986 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614991 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.614997 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615001 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.615005 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.615010 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.615014 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.615018 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615022 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615027 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.615031 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.615035 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.615039 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.615043 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.615048 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.615052 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.615056 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.615062 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.615066 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.615071 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615075 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.615079 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615083 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.615088 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.615092 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.615096 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615100 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.615105 | controller | 18:59:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.615109 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615113 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.615117 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.615122 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.615126 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.615136 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.615140 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.615159 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.615166 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.615170 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.615174 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.615179 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.615183 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615187 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615191 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615195 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.615200 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.615204 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.615208 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.615213 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615220 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615224 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.615229 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.615233 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.615237 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.615241 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.615246 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.615250 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.615254 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.615258 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.615267 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.615274 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615280 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.615286 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615292 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.615298 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.615304 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.615310 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615318 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.615322 | controller | 18:59:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.615327 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615331 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.615335 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.615339 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.615344 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.615348 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.615361 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.615366 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.615371 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.615375 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.615379 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.615383 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.615388 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615394 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615398 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615403 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.615407 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.615411 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.615415 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.615420 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615426 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615430 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.615434 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.615439 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.615443 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.615447 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.615451 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.615456 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.615460 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.615464 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.615468 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.615473 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615477 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.615481 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615486 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.615490 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.615494 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.615498 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615503 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.615507 | controller | 18:59:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.615511 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615516 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.615520 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.615524 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.615530 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.615534 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.615539 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.615548 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.615553 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.615557 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.615562 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.615566 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.615570 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615576 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615580 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615585 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.615589 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.615593 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.615597 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.615602 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615606 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615610 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.615614 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.615619 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.615623 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.615627 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.615631 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.615636 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.615640 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.615644 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.615648 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.615653 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615657 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.615661 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615665 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.615670 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.615674 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.615680 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615684 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.615688 | controller | 18:59:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.615692 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615697 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.615701 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.615705 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.615711 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.615715 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.615719 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.615724 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.615728 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.615738 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.615743 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.615747 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.615751 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615756 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615760 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615764 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.615768 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.615773 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.615777 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.615781 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615785 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.615789 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.615794 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.615798 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.615802 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.615819 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.615824 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.615828 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.615832 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.615839 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.615843 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.615847 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615851 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.615856 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.615860 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.615866 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.615872 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.615878 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615884 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.615890 | controller | 18:59:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.615896 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.615902 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.615908 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.615912 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.615916 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.615921 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.615925 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.615929 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.615933 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.615938 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.615949 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.615953 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.615958 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615963 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615969 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.615975 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.615981 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.615987 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.615992 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.615998 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616007 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616013 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.616017 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.616021 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.616025 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.616030 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.616034 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.616038 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.616042 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.616047 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.616051 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.616055 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616059 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.616063 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616068 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.616073 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.616079 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.616085 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.616091 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.616097 | controller | 18:59:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.616103 | controller | \"ac-watcher\" not found\n logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.616109 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.616115 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.616121 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.616126 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.616132 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.616136 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.616141 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.616176 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.616183 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.616189 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.616195 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.616213 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616223 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616229 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616237 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.616243 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.616249 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.616254 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.616258 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616262 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616267 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.616273 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.616280 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.616286 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.616292 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.616298 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.616304 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.616309 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.616315 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.616321 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.616327 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616333 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.616339 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616344 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.616350 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.616356 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.616362 | controller | expected application_credential_id\"]\n logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.616368 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.616374 | controller | 18:59:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.616380 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.616386 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.616392 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.616398 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.616409 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.616417 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.616423 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.616429 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.616435 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.616441 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.616447 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.616452 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.616458 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616475 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616482 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616488 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.616494 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.616499 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.616505 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.616511 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616517 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616526 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.616532 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.616537 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.616543 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.616549 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.616555 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.616561 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.616567 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.616573 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.616579 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.616585 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616591 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.616599 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616605 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.616611 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.616617 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.616625 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:00 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.616631 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.616637 | controller | 19:00:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.616643 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.616649 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.616654 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.616658 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.616663 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.616667 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.616671 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.616675 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.616680 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.616684 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.616688 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.616692 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.616696 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616701 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616705 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616717 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.616722 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.616726 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.616731 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.616735 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616739 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616744 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.616748 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.616752 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.616756 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.616761 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.616765 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.616769 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.616773 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.616780 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.616786 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.616791 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616795 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.616799 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616804 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.616808 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.616812 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.616816 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.616821 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.616825 | controller | 19:00:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.616829 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.616833 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.616838 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.616842 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.616846 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.616850 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.616855 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.616859 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.616863 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.616867 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.616872 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.616877 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.616882 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616886 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616890 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.616895 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.616905 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.616910 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.616914 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.616918 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616924 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.616929 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.616933 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.616937 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.616941 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.616946 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.616950 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.616954 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.616958 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.616962 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.616967 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.616971 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616975 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.616980 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.616984 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.616988 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.616992 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.616996 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617001 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.617005 | controller | 19:00:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.617009 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617014 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.617018 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.617022 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.617026 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.617031 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.617035 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.617039 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.617043 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.617048 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.617052 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.617056 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.617060 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617066 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617071 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617075 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.617079 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.617090 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.617095 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.617099 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617104 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617108 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.617112 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.617117 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.617121 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.617125 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.617129 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.617133 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.617140 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.617156 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.617164 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.617169 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617173 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.617177 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617181 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.617186 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.617190 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.617194 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617198 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.617203 | controller | 19:00:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.617207 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617211 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.617215 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.617220 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.617228 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.617233 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.617237 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.617241 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.617245 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.617250 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.617254 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.617258 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.617262 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617267 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617271 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617275 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.617279 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.617284 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.617288 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.617300 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617304 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617309 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.617313 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.617317 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.617322 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.617326 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.617330 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.617334 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.617338 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.617343 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.617347 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.617351 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617355 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.617360 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617364 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.617368 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.617372 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.617379 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617383 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.617387 | controller | 19:00:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.617391 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617396 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.617400 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.617404 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.617409 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.617413 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.617417 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.617421 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.617426 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.617430 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.617436 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.617441 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.617445 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617449 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617454 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617458 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.617462 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.617466 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.617470 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.617475 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617488 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617493 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.617497 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.617501 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.617505 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.617510 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.617514 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.617518 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.617522 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.617528 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.617533 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.617537 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617541 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.617546 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617550 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.617554 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.617558 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.617562 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617567 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.617571 | controller | 19:00:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.617575 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617579 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.617584 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.617588 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.617592 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.617596 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.617601 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.617605 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.617609 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.617613 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.617618 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.617622 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.617626 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617630 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617635 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617640 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.617645 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.617649 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.617653 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.617657 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617663 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617674 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.617678 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.617683 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.617687 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.617691 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.617695 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.617701 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.617706 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.617710 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.617714 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.617718 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617723 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.617727 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617731 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.617735 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.617739 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.617744 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:07 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617748 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.617752 | controller | 19:00:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.617756 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617761 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.617765 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.617769 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.617773 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.617778 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.617782 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.617786 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.617790 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.617794 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.617799 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.617803 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.617807 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617813 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617817 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617822 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.617826 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.617830 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.617835 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.617839 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617843 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.617847 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.617858 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.617863 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.617867 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.617872 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.617876 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.617880 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.617884 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.617889 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.617893 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.617897 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617901 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.617906 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.617910 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.617914 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.617918 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.617923 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617927 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.617931 | controller | 19:00:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.617936 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.617942 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.617946 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.617951 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.617956 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.617961 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.617965 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.617969 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.617974 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.617980 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.617984 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.617988 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.617993 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.617997 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618001 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618005 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.618010 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.618014 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.618018 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.618022 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618027 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618031 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.618035 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.618039 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.618049 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.618053 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.618058 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.618062 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.618066 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.618071 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.618076 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.618081 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618085 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.618089 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618093 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.618098 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.618102 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.618108 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618112 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.618116 | controller | 19:00:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.618120 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618125 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.618129 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.618133 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.618138 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.618142 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.618160 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.618164 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.618169 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.618173 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.618177 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.618181 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.618186 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618190 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618194 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618198 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.618202 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.618207 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.618211 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.618215 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618219 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618224 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.618228 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.618232 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.618236 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.618242 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.618252 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.618257 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.618261 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.618268 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.618272 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.618276 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618280 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.618285 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618289 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.618293 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.618297 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.618302 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618306 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.618310 | controller | 19:00:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.618315 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618319 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.618323 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.618327 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.618331 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.618336 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.618340 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.618344 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.618348 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.618352 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.618357 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.618361 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.618365 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618369 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618374 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618378 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.618382 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.618386 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.618391 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.618395 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618402 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618407 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.618411 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.618415 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.618419 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.618424 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.618428 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.618437 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.618442 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.618446 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.618450 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.618454 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618459 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.618463 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618467 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.618471 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.618476 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.618480 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618484 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.618488 | controller | 19:00:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.618493 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618497 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.618501 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.618505 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.618512 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.618517 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.618521 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.618525 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.618531 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.618535 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.618540 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.618544 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.618548 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618554 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618558 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618563 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.618567 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.618571 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.618576 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.618580 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618584 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618588 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.618593 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.618597 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.618601 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.618605 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.618610 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.618614 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.618624 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.618628 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.618633 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.618637 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618641 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.618645 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618650 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.618654 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.618658 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.618662 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618669 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.618673 | controller | 19:00:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.618678 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618682 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.618686 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.618690 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.618696 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.618700 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.618705 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.618709 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.618713 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.618717 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.618722 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.618727 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.618732 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618738 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618744 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618750 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.618756 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.618762 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.618767 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.618773 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618779 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618785 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.618791 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.618795 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.618802 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.618806 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.618811 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.618815 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.618819 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.618823 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.618835 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.618839 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618843 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.618848 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.618852 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.618856 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.618860 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.618868 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:14 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618872 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.618876 | controller | 19:00:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.618880 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.618885 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.618889 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.618893 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.618897 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.618902 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.618906 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.618910 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.618914 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.618919 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.618923 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.618927 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.618931 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618936 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618940 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.618946 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.618950 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.618954 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.618959 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.618963 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618967 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.618971 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.618976 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.618980 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.618984 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.618988 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.618992 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.618997 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.619001 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.619007 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.619011 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.619021 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619025 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.619030 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619034 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.619038 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.619042 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.619047 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619051 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.619055 | controller | 19:00:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.619059 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619064 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.619068 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.619074 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.619079 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.619083 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.619087 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.619091 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.619096 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.619100 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.619104 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.619108 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.619113 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619117 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619121 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619125 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.619129 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.619134 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.619138 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.619142 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619164 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619168 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.619173 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.619177 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.619181 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.619186 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.619190 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.619194 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.619198 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.619203 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.619207 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.619211 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619221 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.619226 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619230 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.619234 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.619238 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.619243 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619247 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.619251 | controller | 19:00:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.619255 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619260 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.619264 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.619268 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.619272 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.619277 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.619281 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.619285 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.619289 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.619293 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.619298 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.619302 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.619306 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619313 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619317 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619322 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.619326 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.619330 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.619334 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.619338 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619345 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619349 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.619355 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.619362 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.619369 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.619375 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.619381 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.619387 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.619393 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.619398 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.619407 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.619413 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619418 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.619424 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619439 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.619444 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.619448 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.619453 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619457 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.619465 | controller | 19:00:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.619469 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619473 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.619477 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.619482 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.619488 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.619492 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.619497 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.619501 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.619505 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.619509 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.619513 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.619518 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.619522 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619526 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619530 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619534 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.619539 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.619543 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.619547 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.619551 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619556 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619560 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.619564 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.619568 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.619573 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.619577 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.619581 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.619585 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.619589 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.619594 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.619598 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.619602 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619606 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.619611 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619615 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.619626 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.619631 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.619637 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619641 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.619646 | controller | 19:00:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.619650 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619654 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.619660 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.619665 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.619669 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.619673 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.619678 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.619682 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.619686 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.619690 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.619694 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.619699 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.619703 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619707 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619711 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619716 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.619720 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.619724 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.619728 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.619733 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619737 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619741 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.619745 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.619750 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.619754 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.619758 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.619762 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.619766 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.619771 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.619777 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.619781 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.619785 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619790 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.619794 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.619798 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.619815 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.619827 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.619834 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619840 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.619846 | controller | 19:00:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.619852 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.619858 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.619864 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.619869 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.619879 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.619885 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.619890 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.619896 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.619902 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.619907 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.619913 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.619919 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.619924 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619930 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619935 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.619941 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.619946 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.619952 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.619958 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.619963 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619973 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.619981 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.619987 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.619993 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.619999 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.620005 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.620011 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.620017 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.620022 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.620028 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.620034 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.620039 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620045 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.620051 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620056 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.620062 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.620067 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.620074 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620091 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.620098 | controller | 19:00:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.620103 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620109 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.620115 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.620120 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.620126 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.620131 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.620137 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.620143 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.620168 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.620174 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.620180 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.620186 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.620192 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620202 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620209 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620215 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.620220 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.620226 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.620232 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.620237 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.620243 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.620248 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.620254 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.620260 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.620266 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.620271 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.620277 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.620283 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.620289 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.620295 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.620301 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.620307 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620313 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.620319 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620324 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.620330 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.620336 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.620342 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:22 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620347 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.620365 | controller | 19:00:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.620373 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620383 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.620389 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.620395 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.620404 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.620408 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.620412 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.620417 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.620421 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.620425 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.620429 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.620434 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.620439 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620445 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620451 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620460 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.620466 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.620472 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.620478 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.620484 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.620492 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.620499 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.620504 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.620510 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.620515 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.620519 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.620524 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.620528 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.620532 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.620536 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.620541 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.620545 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620549 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.620553 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620558 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.620562 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.620566 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.620573 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620578 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.620584 | controller | 19:00:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.620609 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620621 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.620626 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.620630 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.620634 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.620639 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.620643 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.620647 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.620652 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.620656 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.620660 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.620664 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.620669 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620673 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620677 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620681 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.620689 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.620694 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.620698 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.620702 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.620707 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.620711 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.620715 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.620719 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.620724 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.620728 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.620732 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.620737 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.620741 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.620749 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.620754 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.620758 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620762 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.620767 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620771 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.620775 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.620779 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.620784 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620788 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.620792 | controller | 19:00:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.620797 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620801 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.620813 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.620819 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.620825 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.620831 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.620838 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.620843 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.620849 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.620856 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.620861 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.620867 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.620872 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620877 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620881 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.620885 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.620889 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.620894 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.620898 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.620902 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.620909 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.620913 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.620918 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.620922 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.620926 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.620931 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.620935 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.620939 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.620943 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.620948 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.620954 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.620959 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620963 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.620967 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.620972 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.620976 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.620980 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.620984 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.620989 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.620993 | controller | 19:00:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.620999 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621003 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.621008 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.621021 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.621026 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.621032 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.621036 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.621041 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.621045 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.621049 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.621054 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.621058 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.621062 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621068 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621073 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621077 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.621081 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.621086 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.621090 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.621094 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621099 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621103 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.621107 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.621112 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.621116 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.621120 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.621125 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.621129 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.621133 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.621137 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.621142 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.621163 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621168 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.621172 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621177 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.621181 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.621185 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.621189 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621194 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.621198 | controller | 19:00:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.621202 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621206 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.621211 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.621215 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.621221 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.621233 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.621237 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.621242 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.621246 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.621250 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.621254 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.621259 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.621263 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621267 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621273 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621278 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.621282 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.621286 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.621290 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.621295 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621299 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621303 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.621307 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.621312 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.621316 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.621320 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.621324 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.621329 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.621333 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.621337 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.621341 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.621345 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621350 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.621354 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621358 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.621362 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.621367 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.621372 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:28 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621377 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.621381 | controller | 19:00:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.621385 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621390 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.621394 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.621398 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.621404 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.621408 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.621419 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.621423 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.621428 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.621432 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.621437 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.621441 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.621445 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621449 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621453 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621458 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.621462 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.621466 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.621470 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.621474 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621480 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621485 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.621489 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.621493 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.621497 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.621502 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.621506 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.621510 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.621514 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.621520 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.621524 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.621529 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621533 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.621537 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621541 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.621546 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.621551 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.621556 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621562 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.621566 | controller | 19:00:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.621571 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621575 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.621579 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.621583 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.621588 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.621592 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.621596 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.621607 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.621611 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.621615 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.621620 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.621624 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.621628 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621633 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621637 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621641 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.621645 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.621649 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.621654 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.621658 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621664 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621668 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.621672 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.621677 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.621681 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.621685 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.621689 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.621694 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.621698 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.621702 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.621706 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.621710 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621715 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.621719 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621723 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.621727 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.621732 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.621736 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621740 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.621744 | controller | 19:00:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.621749 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621753 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.621757 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.621761 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.621765 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.621770 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.621774 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.621778 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.621782 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.621793 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.621797 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.621801 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.621806 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621815 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621819 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621825 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.621829 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.621833 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.621837 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.621842 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621846 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.621850 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.621854 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.621858 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.621863 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.621867 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.621871 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.621875 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.621880 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.621884 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.621888 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.621892 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621896 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.621901 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.621905 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.621909 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.621913 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.621918 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621922 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.621926 | controller | 19:00:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.621930 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.621935 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.621939 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.621943 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.621949 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.621953 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.621958 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.621962 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.621966 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.621970 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.621980 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.621984 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.621989 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621993 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.621997 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622001 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.622006 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.622010 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.622014 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.622018 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622023 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622027 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.622031 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.622035 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.622040 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.622044 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.622048 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.622052 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.622056 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.622061 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.622065 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.622071 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.622075 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.622080 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.622084 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.622090 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.622094 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.622100 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.622104 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.622108 | controller | 19:00:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.622113 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.622117 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.622121 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.622125 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.622130 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.622134 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.622138 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.622142 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.622161 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.622166 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.622170 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.622174 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.622184 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622191 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622197 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622203 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.622208 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.622214 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.622220 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.622227 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622232 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622238 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.622244 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.622249 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.622255 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.622261 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.622267 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.622272 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.622278 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.622288 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.622302 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.622308 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.622314 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.622320 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.622326 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.622331 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.622336 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.622340 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.622344 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.622348 | controller | 19:00:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.622353 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.622357 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.622361 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.622365 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.622370 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.622374 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.622380 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.622386 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.622392 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.622398 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.622403 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.622409 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.622416 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622430 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622435 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622441 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.622447 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.622453 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.622459 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.622465 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622477 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622483 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.622489 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.622495 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.622501 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.622506 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.622512 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.622518 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.622550 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.622558 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.622564 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.622569 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.622573 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.622578 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.622582 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.622586 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.622590 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.622595 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:35 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.622599 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.622603 | controller | 19:00:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.622607 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.622612 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.622616 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.622620 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.622624 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.622629 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.622633 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.622637 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.622641 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.622645 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.622650 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.622654 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.622658 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622665 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622669 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622681 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.622686 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.622690 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.622694 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.622699 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622703 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622707 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.622712 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.622716 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.622720 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.622724 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.622729 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.622733 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.622737 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.622741 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.622745 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.622750 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.622756 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.622763 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.622769 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.622775 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.622781 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.622787 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.622793 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.622799 | controller | 19:00:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.622805 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.622811 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.622817 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.622823 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.622835 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.622841 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.622847 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.622852 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.622858 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.622863 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.622869 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.622875 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.622880 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622886 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622891 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.622897 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.622913 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.622919 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.622925 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.622931 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622936 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.622942 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.622948 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.622953 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.622959 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.622965 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.622970 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.622976 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.622982 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.622988 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.622994 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.623000 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623005 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.623011 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.623022 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.623028 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.623038 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623044 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.623050 | controller | 19:00:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.623056 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623062 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.623067 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.623073 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.623079 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.623084 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.623090 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.623096 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.623102 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.623108 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.623114 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.623124 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.623131 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623137 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623143 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623177 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.623183 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.623200 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.623207 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.623213 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.623219 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.623225 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.623232 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.623238 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.623244 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.623251 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.623257 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.623264 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.623270 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.623280 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.623287 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.623293 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623300 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.623306 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623312 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.623318 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.623325 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.623332 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623338 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.623344 | controller | 19:00:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.623350 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623355 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.623361 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.623368 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.623373 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.623379 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.623385 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.623391 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.623397 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.623403 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.623408 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.623414 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.623420 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623426 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623432 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623442 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.623448 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.623453 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.623471 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.623478 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.623488 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.623494 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.623500 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.623506 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.623512 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.623518 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.623523 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.623529 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.623539 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.623545 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.623551 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.623557 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623563 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.623569 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623575 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.623580 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.623584 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.623588 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623595 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.623599 | controller | 19:00:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.623603 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623607 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.623612 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.623616 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.623620 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.623624 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.623629 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.623633 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.623637 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.623641 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.623645 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.623650 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.623654 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623661 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623665 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623670 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.623674 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.623678 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.623682 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.623687 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.623701 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.623706 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.623710 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.623715 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.623719 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.623723 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.623727 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.623732 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.623736 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.623740 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.623744 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.623749 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623753 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.623757 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623761 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.623766 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.623770 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.623774 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623778 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.623783 | controller | 19:00:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.623787 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623791 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.623812 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.623819 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.623826 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.623830 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.623834 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.623839 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.623843 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.623847 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.623853 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.623858 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.623862 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623866 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623870 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.623875 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.623879 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.623883 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.623887 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.623892 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.623896 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.623908 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.623913 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.623917 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.623921 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.623925 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.623930 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.623934 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.623938 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.623942 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.623948 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.623953 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623957 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.623961 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.623966 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.623970 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.623974 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.623981 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:42 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623985 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.623990 | controller | 19:00:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.623994 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.623998 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.624002 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.624006 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.624011 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.624015 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.624019 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.624023 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.624028 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.624032 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.624036 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.624040 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.624044 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624049 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624053 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624057 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.624061 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.624066 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.624070 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.624074 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624078 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624083 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.624094 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.624098 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.624103 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.624107 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.624111 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.624117 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.624121 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.624127 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.624132 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.624136 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624140 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.624160 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624167 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.624172 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.624176 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.624180 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624184 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.624189 | controller | 19:00:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.624193 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624197 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.624201 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.624206 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.624210 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.624214 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.624218 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.624222 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.624227 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.624231 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.624235 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.624239 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.624244 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624248 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624252 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624256 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.624260 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.624265 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.624269 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.624273 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624281 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624285 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.624289 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.624294 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.624306 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.624310 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.624315 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.624319 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.624323 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.624327 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.624332 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.624336 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624340 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.624345 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624349 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.624353 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.624357 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.624361 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624366 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.624370 | controller | 19:00:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.624374 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624378 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.624383 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.624387 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.624393 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.624398 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.624402 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.624406 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.624410 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.624416 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.624420 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.624425 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.624429 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624435 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624439 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624444 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.624448 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.624452 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.624456 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.624461 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624465 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624469 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.624473 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.624478 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.624482 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.624493 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.624498 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.624502 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.624506 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.624511 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.624515 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.624519 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624523 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.624527 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624532 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.624536 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.624540 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.624544 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624549 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.624553 | controller | 19:00:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.624557 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624561 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.624566 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.624570 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.624577 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.624581 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.624586 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.624590 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.624594 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.624598 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.624603 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.624607 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.624611 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624615 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624620 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624624 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.624628 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.624632 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.624636 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.624641 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624646 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624651 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.624655 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.624659 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.624663 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.624669 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.624673 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.624684 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.624688 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.624693 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.624697 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.624701 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624706 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.624710 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624714 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.624718 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.624722 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.624729 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624733 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.624737 | controller | 19:00:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.624741 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624746 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.624750 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.624754 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.624758 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.624763 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.624767 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.624771 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.624775 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.624780 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.624784 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.624788 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.624792 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624796 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624801 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624807 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.624811 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.624815 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.624819 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.624824 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624828 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.624832 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.624836 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.624842 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.624847 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.624851 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.624855 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.624859 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.624870 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.624876 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.624881 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.624885 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624889 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.624893 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.624898 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.624902 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.624906 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.624910 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:48 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624914 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.624919 | controller | 19:00:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.624923 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.624927 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.624931 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.624936 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.624940 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.624944 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.624948 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.624953 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.624958 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.624963 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.624967 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.624971 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.624975 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624980 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624984 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.624988 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.624992 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.624997 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.625001 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.625005 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625011 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625015 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.625019 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.625024 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.625028 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.625032 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.625036 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.625041 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.625045 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.625049 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.625059 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.625063 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625068 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.625072 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625076 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.625081 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.625085 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.625089 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625093 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.625098 | controller | 19:00:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.625102 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625106 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.625110 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.625114 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.625119 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.625123 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.625127 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.625131 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.625136 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.625140 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.625160 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.625169 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.625173 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625180 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625184 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625188 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.625193 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.625197 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.625201 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.625207 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625213 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625219 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.625225 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.625231 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.625238 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.625242 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.625247 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.625251 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.625255 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.625259 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.625266 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.625280 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625285 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.625289 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625293 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.625297 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.625302 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.625306 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625310 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.625314 | controller | 19:00:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.625319 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625323 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.625327 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.625331 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.625338 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.625342 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.625346 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.625350 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.625355 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.625359 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.625363 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.625367 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.625372 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625376 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625380 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625384 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.625388 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.625393 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.625397 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.625401 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625405 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625410 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.625414 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.625418 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.625422 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.625426 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.625431 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.625435 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.625439 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.625443 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.625448 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.625452 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625462 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.625467 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625471 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.625475 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.625479 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.625486 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625490 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.625494 | controller | 19:00:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.625498 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625503 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.625507 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.625513 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.625517 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.625521 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.625526 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.625530 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.625534 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.625538 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.625542 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.625547 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.625551 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625555 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625559 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625564 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.625568 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.625572 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.625576 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.625580 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625586 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625591 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.625595 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.625599 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.625603 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.625607 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.625612 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.625616 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.625620 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.625626 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.625630 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.625635 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625639 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.625649 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625654 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.625658 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.625662 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.625667 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625671 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.625675 | controller | 19:00:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.625680 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625684 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.625688 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.625692 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.625699 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.625705 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.625709 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.625713 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.625718 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.625722 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.625726 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.625730 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.625734 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625739 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625743 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625747 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.625751 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.625756 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.625760 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.625764 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625770 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625774 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.625779 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.625783 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.625787 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.625791 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.625796 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.625801 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.625806 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.625810 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.625814 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.625818 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625823 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.625827 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.625831 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.625842 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.625847 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.625851 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:54 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625858 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.625864 | controller | 19:00:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.625870 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.625875 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.625880 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.625884 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.625888 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.625892 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.625896 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.625901 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.625905 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.625909 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.625913 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.625918 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.625922 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625928 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625932 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.625937 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.625941 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.625945 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.625949 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.625954 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625958 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.625962 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.625966 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.625970 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.625975 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.625979 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.625983 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.625987 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.625992 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.625996 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.626000 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.626004 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.626008 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.626013 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.626017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.626021 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.626032 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.626037 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.626041 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.626046 | controller | 19:00:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.626050 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.626054 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.626061 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.626065 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.626072 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.626076 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.626080 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.626085 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.626089 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.626093 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.626097 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.626102 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.626106 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.626110 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.626114 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.626120 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.626124 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.626129 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.626133 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.626137 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.626141 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.626160 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.626164 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.626169 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.626173 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.626177 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.626181 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.626186 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.626190 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.626194 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.626198 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.626203 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.626207 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.626211 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.626215 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.626219 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.626224 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.626231 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.626242 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.626247 | controller | 19:00:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.626251 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.626256 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-09 19:10:03.626260 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-09 19:10:03.626264 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-09 19:10:03.626269 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-09 19:10:03.626273 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-09 19:10:03.626277 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-09 19:10:03.626281 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-09 19:10:03.626286 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-09 19:10:03.626290 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-09 19:10:03.626294 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-09 19:10:03.626298 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-09 19:10:03.626303 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.626307 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.626311 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-09 19:10:03.626315 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-09 19:10:03.626319 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-09 19:10:03.626324 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-09 19:10:03.626328 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-09 19:10:03.626332 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.626337 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-09 19:10:03.626344 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-09 19:10:03.626348 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-09 19:10:03.626353 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-09 19:10:03.626357 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-09 19:10:03.626361 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-09 19:10:03.626365 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-09 19:10:03.626370 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-09 19:10:03.626374 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-09 19:10:03.626380 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-09 19:10:03.626384 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-09 19:10:03.626388 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.626392 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-09 19:10:03.626397 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-09 19:10:03.626401 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.626405 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-09 19:10:03.626409 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-09 19:10:03.626413 | controller | expected application_credential_id\"]\n logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.626418 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-09 19:10:03.626428 | controller | 19:00:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-09 19:10:03.626433 | controller | \"ac-watcher\" not found\n logger.go:42: 19:00:59 | appcred-tests/2-deploy-appcred 2026-03-09 19:10:03.626437 | controller | | test step failed 2-deploy-appcred\n case.go:396: failed in step 2-deploy-appcred\n 2026-03-09 19:10:03.626442 | controller | \ case.go:398: command \"NS=\\\"${NAMESPACE}\\\"\\\\n echo \\\"Waiting for KeystoneApplicationCredential...\" 2026-03-09 19:10:03.626446 | controller | failed, exit status 1\n logger.go:42: 19:00:59 | appcred-tests | skipping kubernetes 2026-03-09 19:10:03.626450 | controller | event logging\n=== CONT kuttl/harness/watcher-notification\n logger.go:42: 19:00:59 2026-03-09 19:10:03.626454 | controller | | watcher-notification | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-09 19:10:03.626459 | controller | \ logger.go:42: 19:00:59 | watcher-notification/0-cleanup-watcher | starting test 2026-03-09 19:10:03.626463 | controller | step 0-cleanup-watcher\n logger.go:42: 19:00:59 | watcher-notification/0-cleanup-watcher 2026-03-09 19:10:03.626467 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626471 | controller | | starting test step 1-deploy-with-notification\n logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626476 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:00:59 2026-03-09 19:10:03.626480 | controller | | watcher-notification/1-deploy-with-notification | running command: [sh -c set 2026-03-09 19:10:03.626484 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.626488 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.626493 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.626497 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep 2026-03-09 19:10:03.626501 | controller | -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.626506 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.626510 | controller | 19:00:59 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626514 | controller | \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | Now 2026-03-09 19:10:03.626520 | controller | using project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.626524 | controller | \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626529 | controller | head -1\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626533 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626537 | controller | | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.626541 | controller | \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | + 2026-03-09 19:10:03.626546 | controller | APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626550 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626556 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.626560 | controller | \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.626566 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.626570 | controller | \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | error: 2026-03-09 19:10:03.626574 | controller | Internal error occurred: error executing command in container: container is not 2026-03-09 19:10:03.626579 | controller | created or running\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626583 | controller | | ++ echo\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626587 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626591 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626595 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.626601 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.626605 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.626615 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.626619 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.626624 | controller | 19:01:02 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626628 | controller | \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.626632 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.626636 | controller | \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626641 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.626645 | controller | \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626649 | controller | head -1\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626653 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626658 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626662 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626666 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.626672 | controller | \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.626676 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.626681 | controller | \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | Error 2026-03-09 19:10:03.626685 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.626689 | controller | 19:01:02 | watcher-notification/1-deploy-with-notification | ++ echo\n logger.go:42: 2026-03-09 19:10:03.626693 | controller | 19:01:02 | watcher-notification/1-deploy-with-notification | + '[' 0 == 1 ']'\n 2026-03-09 19:10:03.626698 | controller | \ logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | running 2026-03-09 19:10:03.626702 | controller | command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc 2026-03-09 19:10:03.626706 | controller | get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.626710 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.626715 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep 2026-03-09 19:10:03.626719 | controller | -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.626723 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.626727 | controller | 19:01:03 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626731 | controller | \ logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.626736 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.626740 | controller | \ logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626744 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.626748 | controller | \ logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626753 | controller | head -1\n logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626757 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626761 | controller | | + APIPOD=\n logger.go:42: 19:01:04 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626765 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626769 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.626774 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.626778 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.626782 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.626786 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.626791 | controller | 19:01:04 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626795 | controller | \ logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.626804 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.626809 | controller | \ logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626813 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.626819 | controller | \ logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626824 | controller | head -1\n logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626828 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626832 | controller | | + APIPOD=\n logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626836 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626842 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.626847 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.626851 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.626855 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.626860 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.626864 | controller | 19:01:06 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626868 | controller | \ logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.626872 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.626876 | controller | \ logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626881 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.626885 | controller | \ logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626889 | controller | head -1\n logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626893 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626898 | controller | | + APIPOD=\n logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626902 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626906 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.626910 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.626915 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.626921 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.626927 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.626933 | controller | 19:01:07 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626939 | controller | \ logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.626945 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.626951 | controller | \ logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626957 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.626962 | controller | \ logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.626971 | controller | grep -v '^$'\n logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626977 | controller | | ++ head -1\n logger.go:42: 19:01:08 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626986 | controller | | + APIPOD=\n logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.626992 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.626998 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.627004 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.627010 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.627016 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627022 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627039 | controller | 19:01:09 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627046 | controller | \ logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627052 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627058 | controller | \ logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627064 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627070 | controller | \ logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627076 | controller | head -1\n logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627082 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627088 | controller | | + APIPOD=\n logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627094 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627100 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.627108 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.627115 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.627121 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627129 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627135 | controller | 19:01:10 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627141 | controller | \ logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627174 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627183 | controller | \ logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627188 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627192 | controller | \ logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627196 | controller | head -1\n logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627200 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627205 | controller | | + APIPOD=\n logger.go:42: 19:01:11 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627209 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627216 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.627220 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.627225 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.627229 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627233 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627237 | controller | 19:01:11 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627242 | controller | \ logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627246 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627250 | controller | \ logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627254 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627258 | controller | \ logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627263 | controller | head -1\n logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627267 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627271 | controller | | + APIPOD=\n logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627275 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627280 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.627284 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.627288 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.627302 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627307 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627311 | controller | 19:01:13 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627315 | controller | \ logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627320 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627324 | controller | \ logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627330 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627335 | controller | \ logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627339 | controller | head -1\n logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627343 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627347 | controller | | + APIPOD=\n logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627352 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627356 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.627360 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.627364 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.627370 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627375 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627379 | controller | 19:01:14 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627383 | controller | \ logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627387 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627392 | controller | \ logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627396 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627400 | controller | \ logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627404 | controller | head -1\n logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627409 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:15 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627413 | controller | | + APIPOD=\n logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627417 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627424 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.627428 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.627433 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.627439 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627443 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627447 | controller | 19:01:16 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627452 | controller | \ logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627456 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627460 | controller | \ logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627464 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627468 | controller | \ logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627473 | controller | head -1\n logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627477 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627481 | controller | | + APIPOD=\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627485 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627490 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.627501 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.627505 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.627510 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627514 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627518 | controller | 19:01:17 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627524 | controller | \ logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627529 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627533 | controller | \ logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627537 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627542 | controller | \ logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627546 | controller | head -1\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627550 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627554 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627559 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627563 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.627567 | controller | \ logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.627571 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.627575 | controller | \ logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | error: 2026-03-09 19:10:03.627580 | controller | unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-09 19:10:03.627584 | controller | 19:01:18 | watcher-notification/1-deploy-with-notification | ++ echo\n logger.go:42: 2026-03-09 19:10:03.627588 | controller | 19:01:18 | watcher-notification/1-deploy-with-notification | + '[' 0 == 1 ']'\n 2026-03-09 19:10:03.627592 | controller | \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | running 2026-03-09 19:10:03.627597 | controller | command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc 2026-03-09 19:10:03.627601 | controller | get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.627605 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.627609 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep 2026-03-09 19:10:03.627614 | controller | -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627618 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627622 | controller | 19:01:19 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627626 | controller | \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627630 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627635 | controller | \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627639 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627643 | controller | \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627647 | controller | head -1\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627652 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627656 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627660 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627666 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.627672 | controller | \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.627676 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.627681 | controller | \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627693 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.627697 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.627702 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.627706 | controller | = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.627723 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.627728 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.627733 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.627737 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.627741 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.627745 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.627750 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.627754 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.627758 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.627763 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.627767 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.627771 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.627775 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.627780 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.627784 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.627788 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-09 19:10:03.627809 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.627815 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.627819 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.627823 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.627827 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.627834 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.627838 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.627842 | controller | period = 900\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627847 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627851 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627855 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.627859 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.627864 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.627868 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.627872 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.627892 | controller | 19:01:20 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.627897 | controller | \ logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.627901 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.627909 | controller | \ logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627913 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.627917 | controller | \ logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627922 | controller | head -1\n logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627928 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627940 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627946 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.627951 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.627955 | controller | \ logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.627959 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.627963 | controller | \ logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.627968 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.627972 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.627976 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.627981 | controller | = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.627985 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.627989 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628003 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.628007 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628012 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.628016 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.628020 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.628024 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.628029 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.628033 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.628037 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.628043 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.628048 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628052 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.628056 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.628061 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-09 19:10:03.628065 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.628069 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.628073 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.628077 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.628082 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.628086 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.628090 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.628103 | controller | period = 900\n logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628108 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628113 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.628117 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.628121 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.628125 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.628130 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.628134 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.628138 | controller | 19:01:22 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.628142 | controller | \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.628168 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.628175 | controller | \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628180 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.628184 | controller | \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628188 | controller | head -1\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628192 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628196 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628201 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628205 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.628209 | controller | \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.628213 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.628217 | controller | \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628222 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628226 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.628230 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.628234 | controller | = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.628239 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.628243 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628247 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.628251 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628256 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.628260 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.628264 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.628270 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.628276 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.628282 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.628288 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.628294 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.628299 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628313 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.628317 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.628324 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-09 19:10:03.628328 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.628332 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.628337 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.628341 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.628345 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.628350 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.628354 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.628358 | controller | period = 900\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628362 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628367 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.628371 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.628378 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.628383 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.628387 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.628391 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.628395 | controller | 19:01:24 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.628400 | controller | \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.628404 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.628408 | controller | \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628412 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.628417 | controller | \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628421 | controller | head -1\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628425 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628429 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628434 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628438 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.628442 | controller | \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.628447 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.628451 | controller | \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628455 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628463 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.628467 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.628471 | controller | = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.628476 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.628480 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628491 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.628496 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628500 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.628504 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.628509 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.628513 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.628517 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.628522 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.628526 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.628530 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.628534 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628539 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.628543 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.628547 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-09 19:10:03.628551 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.628556 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.628560 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.628567 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.628571 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.628576 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.628580 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.628584 | controller | period = 900\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628588 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628593 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.628597 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.628603 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.628607 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.628612 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.628616 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.628620 | controller | 19:01:26 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.628624 | controller | \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.628629 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.628633 | controller | \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628637 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.628641 | controller | \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628645 | controller | head -1\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628650 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628654 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628658 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628662 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.628667 | controller | \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.628679 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.628684 | controller | \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628688 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628692 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.628697 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.628701 | controller | = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.628705 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.628710 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628714 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.628718 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628722 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.628727 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.628731 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.628737 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.628741 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.628745 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.628750 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.628754 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.628758 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628762 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.628766 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.628771 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-09 19:10:03.628775 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.628779 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.628783 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.628788 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.628792 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.628796 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.628801 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.628805 | controller | period = 900\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628809 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628813 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.628817 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.628822 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.628826 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.628830 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.628834 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.628839 | controller | 19:01:27 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.628848 | controller | \ logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.628853 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.628857 | controller | \ logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628861 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.628866 | controller | \ logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628870 | controller | head -1\n logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628874 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628878 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628884 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.628890 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.628895 | controller | \ logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.628899 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.628903 | controller | \ logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.628907 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628912 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.628916 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.628920 | controller | = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.628924 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.628929 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.628933 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.628937 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628941 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.628946 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.628952 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.628956 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.628960 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.628964 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.628969 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.628973 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.628977 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.628983 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.628987 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.628992 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-09 19:10:03.628996 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.629000 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.629004 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.629009 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.629014 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.629019 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.629030 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.629034 | controller | period = 900\n logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.629039 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.629043 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.629047 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.629052 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.629056 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.629060 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-09 19:10:03.629064 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-09 19:10:03.629069 | controller | 19:01:29 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-09 19:10:03.629073 | controller | \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | Already 2026-03-09 19:10:03.629077 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.629081 | controller | \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.629085 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.629090 | controller | \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.629094 | controller | head -1\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.629098 | controller | | ++ grep -v '^$'\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.629102 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.629107 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.629111 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-09 19:10:03.629115 | controller | \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | +++ 2026-03-09 19:10:03.629119 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.629124 | controller | \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ 2026-03-09 19:10:03.629128 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.629132 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.629136 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.629141 | controller | = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.629158 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.629164 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.629172 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.629176 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.629180 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.629185 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.629191 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.629196 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.629200 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.629204 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.629208 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.629218 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.629223 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.629227 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.629232 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.629236 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-09 19:10:03.629240 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.629244 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.629249 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.629253 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.629257 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.629261 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.629266 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.629270 | controller | period = 900\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.629274 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification 2026-03-09 19:10:03.629279 | controller | | test step completed 1-deploy-with-notification\n logger.go:42: 19:01:29 | watcher-notification/2-cleanup-watcher 2026-03-09 19:10:03.629283 | controller | | starting test step 2-cleanup-watcher\n logger.go:42: 19:01:35 | watcher-notification/2-cleanup-watcher 2026-03-09 19:10:03.629287 | controller | | test step completed 2-cleanup-watcher\n logger.go:42: 19:01:35 | watcher-notification 2026-03-09 19:10:03.629291 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-topology\n 2026-03-09 19:10:03.629295 | controller | \ logger.go:42: 19:01:35 | watcher-topology | Skipping creation of user-supplied 2026-03-09 19:10:03.629300 | controller | namespace: watcher-kuttl-default\n logger.go:42: 19:01:35 | watcher-topology/0-cleanup-watcher 2026-03-09 19:10:03.629304 | controller | | starting test step 0-cleanup-watcher\n logger.go:42: 19:01:35 | watcher-topology/0-cleanup-watcher 2026-03-09 19:10:03.629310 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology 2026-03-09 19:10:03.629314 | controller | | starting test step 1-deploy-with-topology\n logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology 2026-03-09 19:10:03.629319 | controller | | Topology:watcher-kuttl-default/watcher-api created\n logger.go:42: 19:01:35 2026-03-09 19:10:03.629325 | controller | | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.629329 | controller | created\n logger.go:42: 19:01:59 | watcher-topology/1-deploy-with-topology | 2026-03-09 19:10:03.629333 | controller | test step completed 1-deploy-with-topology\n logger.go:42: 19:01:59 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629337 | controller | | starting test step 2-cleanup-watcher\n logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629342 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.629346 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.629350 | controller | ]\n ]\n logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | 2026-03-09 19:10:03.629354 | controller | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.629359 | controller | value -c Name -c Type\n logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629363 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:02:02 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629367 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629371 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.629375 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.629380 | controller | ]\n ]\n logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | 2026-03-09 19:10:03.629384 | controller | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.629388 | controller | value -c Name -c Type\n logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629392 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:02:06 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629402 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629406 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.629410 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.629415 | controller | ]\n ]\n logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | 2026-03-09 19:10:03.629419 | controller | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.629423 | controller | value -c Name -c Type\n logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629427 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:02:09 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629431 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:02:09 | watcher-topology/2-cleanup-watcher 2026-03-09 19:10:03.629436 | controller | | test step completed 2-cleanup-watcher\n logger.go:42: 19:02:09 | watcher-topology 2026-03-09 19:10:03.629440 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-tls-certs-change\n 2026-03-09 19:10:03.629444 | controller | \ logger.go:42: 19:02:09 | watcher-tls-certs-change | Skipping creation of user-supplied 2026-03-09 19:10:03.629448 | controller | namespace: watcher-kuttl-default\n logger.go:42: 19:02:09 | watcher-tls-certs-change/0-cleanup-watcher 2026-03-09 19:10:03.629453 | controller | | starting test step 0-cleanup-watcher\n logger.go:42: 19:02:09 | watcher-tls-certs-change/0-cleanup-watcher 2026-03-09 19:10:03.629458 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629463 | controller | | starting test step 1-deploy-with-tlse\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629467 | controller | | Certificate:watcher-kuttl-default/watcher-internal-svc created\n logger.go:42: 2026-03-09 19:10:03.629471 | controller | 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc 2026-03-09 19:10:03.629475 | controller | created\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629480 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:02:09 2026-03-09 19:10:03.629484 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo 2026-03-09 19:10:03.629488 | controller | pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.629492 | controller | cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE 2026-03-09 19:10:03.629497 | controller | secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n 2026-03-09 19:10:03.629503 | controller | \ # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" 2026-03-09 19:10:03.629507 | controller | != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-09 19:10:03.629511 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-09 19:10:03.629516 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.629520 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.629524 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-09 19:10:03.629528 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629533 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.629537 | controller | \ logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.629543 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.629547 | controller | 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.629551 | controller | 19:02:10 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.629555 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629560 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.629564 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629568 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629572 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629577 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629581 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.629585 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629594 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629599 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629603 | controller | 1\n fi\n ]\n logger.go:42: 19:02:10 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629610 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.629615 | controller | \ logger.go:42: 19:02:11 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.629619 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.629623 | controller | 19:02:11 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.629627 | controller | 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.629632 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629636 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.629640 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629644 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629649 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629653 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629657 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.629661 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629665 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629670 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629674 | controller | 1\n fi\n ]\n logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629678 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.629682 | controller | \ logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.629687 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.629691 | controller | 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.629695 | controller | 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.629699 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629704 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.629708 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629712 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629716 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629721 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629725 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.629729 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629735 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629741 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629747 | controller | 1\n fi\n ]\n logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629753 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.629762 | controller | \ logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.629766 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.629770 | controller | 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.629775 | controller | 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.629779 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629783 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.629787 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629798 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629802 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629807 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629811 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.629815 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629819 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629824 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629828 | controller | 1\n fi\n ]\n logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629832 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.629836 | controller | \ logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.629843 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.629847 | controller | 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.629852 | controller | 19:02:15 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.629856 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629860 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.629864 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629870 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629876 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629882 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629888 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.629894 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629899 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629904 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629908 | controller | 1\n fi\n ]\n logger.go:42: 19:02:15 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629912 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.629916 | controller | \ logger.go:42: 19:02:16 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.629923 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.629927 | controller | 19:02:16 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.629932 | controller | 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.629936 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629940 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.629944 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629949 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629953 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629959 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.629963 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.629967 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.629972 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.629976 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.629982 | controller | 1\n fi\n ]\n logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.629987 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.629991 | controller | \ logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.629995 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.630000 | controller | 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.630011 | controller | 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.630015 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630020 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.630024 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630028 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630032 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630036 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630042 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.630046 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630051 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630055 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630059 | controller | 1\n fi\n ]\n logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630064 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.630068 | controller | \ logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.630072 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.630078 | controller | 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.630082 | controller | 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.630087 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630091 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.630095 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630099 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630103 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630108 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630112 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.630116 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630120 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630125 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630129 | controller | 1\n fi\n ]\n logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630133 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.630137 | controller | \ logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.630142 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.630161 | controller | 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.630166 | controller | 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.630170 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630174 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.630178 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630185 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630189 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630193 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630197 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.630202 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630206 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630210 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630214 | controller | 1\n fi\n ]\n logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630225 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.630229 | controller | \ logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.630234 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-09 19:10:03.630238 | controller | 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.630242 | controller | 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.630248 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630253 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.630257 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630261 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630266 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630270 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630274 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.630278 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630283 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630287 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630291 | controller | 1\n fi\n ]\n logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630295 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.630300 | controller | \ logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | error: 2026-03-09 19:10:03.630304 | controller | unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-09 19:10:03.630308 | controller | 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.630312 | controller | 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.630317 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630321 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.630325 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630329 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630334 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630338 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630342 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.630346 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630351 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630355 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630359 | controller | 1\n fi\n ]\n logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630363 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.630368 | controller | \ logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | error: 2026-03-09 19:10:03.630372 | controller | unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-09 19:10:03.630376 | controller | 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-09 19:10:03.630380 | controller | 19:02:24 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-09 19:10:03.630385 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630392 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-09 19:10:03.630396 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630401 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630405 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630409 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-09 19:10:03.630418 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-09 19:10:03.630423 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.630427 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.630431 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-09 19:10:03.630436 | controller | 1\n fi\n ]\n logger.go:42: 19:02:24 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630440 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.630444 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-09 19:10:03.630449 | controller | CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630453 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.630457 | controller | 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.630461 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.630466 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.630470 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.630474 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.630478 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.630483 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.630487 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.630493 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.630497 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.630502 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.630506 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.630510 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.630514 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.630520 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.630525 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.630529 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.630533 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.630538 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.630542 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.630546 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630550 | controller | | ++ base64 --decode\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630555 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.630564 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-09 19:10:03.630570 | controller | CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630574 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.630578 | controller | 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.630583 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.630587 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.630591 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.630596 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.630600 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.630607 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.630611 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.630616 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.630620 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.630624 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.630628 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.630633 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.630639 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.630643 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.630647 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.630651 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.630656 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.630660 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.630664 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.630669 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630673 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630677 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.630681 | controller | 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.630686 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.630690 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.630694 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.630704 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.630708 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.630713 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.630717 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.630721 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.630725 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.630732 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.630736 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.630740 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.630744 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.630750 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.630755 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.630759 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.630763 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.630767 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.630772 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.630776 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 2026-03-09 19:10:03.630780 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.630785 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.630789 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.630793 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.630797 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.630802 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.630806 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.630810 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.630815 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.630819 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.630823 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.630827 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.630837 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.630842 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.630848 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.630852 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.630857 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.630862 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.630867 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.630871 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.630875 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.630880 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630884 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-09 19:10:03.630888 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-09 19:10:03.630893 | controller | CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.630897 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.630901 | controller | 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.630905 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.630910 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.630914 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.630918 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.630922 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.630927 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.630931 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.630935 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.630940 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.630944 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.630948 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.630952 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.630957 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.630961 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.630965 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.630976 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.630984 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.630988 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.630992 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.630997 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631001 | controller | | ++ base64 --decode\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631007 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.631012 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-09 19:10:03.631016 | controller | CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631020 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.631024 | controller | 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.631029 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.631033 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.631037 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.631041 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.631046 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.631050 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.631054 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.631058 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.631063 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.631067 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.631071 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.631075 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.631080 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.631084 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.631091 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.631095 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.631100 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.631104 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.631108 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.631112 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631122 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631127 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.631131 | controller | 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.631135 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.631142 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.631159 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.631164 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.631168 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.631172 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.631177 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.631181 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.631185 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.631189 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.631194 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.631198 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.631202 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.631206 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.631211 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.631215 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.631221 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.631225 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.631230 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.631234 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 2026-03-09 19:10:03.631238 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.631242 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.631247 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.631251 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.631255 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.631270 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.631280 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.631285 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.631289 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.631294 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.631298 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.631302 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.631306 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.631310 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.631315 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.631319 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.631323 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.631327 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.631332 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.631336 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.631342 | controller | \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.631346 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631351 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-09 19:10:03.631355 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-09 19:10:03.631359 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.631364 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.631368 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-09 19:10:03.631372 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-09 19:10:03.631376 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-09 19:10:03.631380 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.631385 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.631389 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-09 19:10:03.631393 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631397 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.631402 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-09 19:10:03.631406 | controller | CERTIFICATE-----\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631410 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.631414 | controller | 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.631419 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.631423 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.631434 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.631439 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.631443 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.631448 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.631452 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.631456 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.631460 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.631465 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.631472 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.631476 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.631480 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.631485 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.631489 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.631493 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.631498 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.631502 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.631508 | controller | \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.631512 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631516 | controller | | ++ base64 --decode\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631521 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.631525 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-09 19:10:03.631529 | controller | CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631533 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.631538 | controller | 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.631542 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.631546 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.631551 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.631555 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.631559 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.631563 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.631573 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.631577 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.631582 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.631588 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.631594 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.631598 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.631602 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.631607 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.631611 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.631615 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.631619 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.631624 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.631628 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.631632 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631636 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631641 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.631645 | controller | 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.631649 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.631654 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.631658 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.631662 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.631667 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.631673 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.631679 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.631685 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.631690 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.631696 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.631705 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.631711 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.631725 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.631732 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.631742 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.631748 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.631753 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.631759 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.631765 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.631771 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 2026-03-09 19:10:03.631777 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.631783 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.631807 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.631814 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.631820 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.631825 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.631831 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.631837 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.631843 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.631849 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.631855 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.631861 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.631867 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.631872 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.631882 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.631888 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.631894 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.631900 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.631906 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.631915 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.631921 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.631927 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631944 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-09 19:10:03.631951 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-09 19:10:03.631958 | controller | CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.631964 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.631970 | controller | 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.631976 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.631983 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.631989 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.631996 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.632002 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.632008 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.632014 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.632021 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.632027 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.632033 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.632039 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.632045 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.632054 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.632061 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.632067 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.632073 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.632079 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.632086 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.632092 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.632098 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632105 | controller | | ++ base64 --decode\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632111 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.632117 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-09 19:10:03.632123 | controller | CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632130 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.632136 | controller | 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.632177 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.632190 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.632197 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.632208 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.632214 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.632220 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.632226 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.632232 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.632239 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.632244 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.632251 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.632260 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.632267 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.632273 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.632279 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.632285 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.632291 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.632297 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.632304 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.632310 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632317 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632323 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.632329 | controller | 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.632335 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.632341 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.632347 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.632353 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.632359 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.632365 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.632386 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.632393 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.632399 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.632406 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.632412 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.632418 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.632424 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.632433 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.632440 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.632446 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.632452 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.632458 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.632464 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.632470 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 2026-03-09 19:10:03.632479 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.632486 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.632492 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.632498 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.632504 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.632511 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.632517 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.632524 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.632530 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.632536 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.632544 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.632551 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.632557 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.632563 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.632569 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.632586 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.632594 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.632605 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.632611 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.632617 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.632624 | controller | \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.632633 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632640 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-09 19:10:03.632647 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-09 19:10:03.632653 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.632659 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.632666 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-09 19:10:03.632672 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-09 19:10:03.632681 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-09 19:10:03.632687 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.632694 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.632700 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-09 19:10:03.632707 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632714 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.632720 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-09 19:10:03.632727 | controller | CERTIFICATE-----\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632734 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.632740 | controller | 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.632747 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.632753 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.632759 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.632765 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.632772 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.632778 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.632785 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.632795 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.632802 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.632809 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.632828 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.632839 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.632846 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.632853 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.632860 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.632867 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.632874 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.632880 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.632887 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.632894 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632901 | controller | | ++ base64 --decode\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632908 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.632914 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-09 19:10:03.632921 | controller | CERTIFICATE-----\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.632928 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.632935 | controller | 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.632941 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.632948 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.632955 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.632962 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.632969 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.632975 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.632985 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.632992 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.632998 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.633005 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.633011 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.633018 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.633024 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.633031 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.633037 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.633058 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.633066 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.633073 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.633080 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.633087 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.633093 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.633099 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.633106 | controller | 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.633112 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.633118 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.633124 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.633130 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.633137 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.633143 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.633167 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.633174 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.633184 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.633190 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.633196 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.633202 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.633208 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.633214 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.633221 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.633227 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.633233 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.633238 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.633245 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.633251 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:28 2026-03-09 19:10:03.633257 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.633263 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.633281 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.633292 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.633299 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.633305 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.633311 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.633318 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.633324 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.633330 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.633336 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.633342 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.633352 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.633358 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.633364 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.633370 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.633377 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.633383 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.633389 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.633396 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.633403 | controller | \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.633409 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.633415 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-09 19:10:03.633422 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-09 19:10:03.633428 | controller | CERTIFICATE-----\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.633435 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.633441 | controller | 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.633447 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.633454 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.633460 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.633467 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.633473 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.633496 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.633503 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.633510 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.633516 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.633525 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.633531 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.633537 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.633543 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.633548 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.633554 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.633560 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.633566 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.633572 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.633578 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.633585 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.633591 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.633597 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.633603 | controller | --decode\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.633609 | controller | | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:29 2026-03-09 19:10:03.633616 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.633622 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.633628 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.633635 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.633641 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.633647 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.633653 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.633659 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.633665 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.633675 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.633681 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.633701 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.633709 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.633716 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.633722 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.633729 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.633736 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.633742 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.633749 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.633755 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.633761 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.633768 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.633774 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.633781 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.633788 | controller | 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.633794 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.633801 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.633808 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.633814 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.633820 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.633826 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.633833 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.633839 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.633845 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.633850 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.633866 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.633872 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.633878 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.633885 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.633891 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.633908 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.633915 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.633922 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.633928 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.633933 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:29 2026-03-09 19:10:03.633939 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.633945 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.633951 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.633957 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.633963 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.633969 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.633975 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.633981 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.633987 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.633993 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.633999 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.634005 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.634011 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.634017 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.634030 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.634036 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.634041 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.634047 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.634053 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.634059 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.634065 | controller | \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.634069 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634077 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-09 19:10:03.634083 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-09 19:10:03.634089 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.634095 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.634113 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-09 19:10:03.634118 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-09 19:10:03.634123 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-09 19:10:03.634127 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.634131 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.634135 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-09 19:10:03.634140 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634157 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.634165 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-09 19:10:03.634169 | controller | CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634173 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.634178 | controller | 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.634182 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.634186 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.634190 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.634195 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.634201 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.634206 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.634210 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.634214 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.634218 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.634223 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.634227 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.634231 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.634235 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.634240 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.634244 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.634251 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.634256 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.634260 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.634264 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.634269 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634280 | controller | | ++ base64 --decode\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634285 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.634289 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-09 19:10:03.634294 | controller | CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634298 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.634302 | controller | 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.634306 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.634311 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.634315 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.634323 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.634327 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.634332 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.634336 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.634340 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.634344 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.634348 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.634353 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.634357 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.634361 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.634365 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.634370 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.634374 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.634378 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.634382 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.634387 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.634391 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634395 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634399 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.634404 | controller | 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.634408 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.634412 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.634425 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.634430 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.634434 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.634441 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.634445 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.634449 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.634453 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.634458 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.634462 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.634466 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.634470 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.634475 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.634479 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.634483 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.634487 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.634491 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.634496 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.634500 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:30 2026-03-09 19:10:03.634504 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.634508 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.634513 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.634517 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.634521 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.634525 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.634530 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.634536 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.634540 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.634546 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.634550 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.634560 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.634565 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.634569 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.634573 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.634577 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.634582 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.634586 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.634590 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.634595 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.634599 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.634603 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634607 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-09 19:10:03.634612 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-09 19:10:03.634618 | controller | CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634623 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.634640 | controller | 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.634647 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.634653 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.634659 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.634665 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.634671 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.634677 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.634683 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.634693 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.634699 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.634709 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.634716 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.634722 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.634728 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.634748 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.634756 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.634763 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.634769 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.634775 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.634781 | controller | \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.634791 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634798 | controller | | ++ base64 --decode\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634804 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.634811 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-09 19:10:03.634817 | controller | CERTIFICATE-----\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634823 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.634830 | controller | 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.634836 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.634842 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.634847 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.634851 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.634856 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.634860 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.634868 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.634873 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.634877 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.634881 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.634886 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.634890 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.634897 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.634901 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.634906 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.634910 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.634914 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.634927 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.634932 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.634937 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634941 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.634945 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.634950 | controller | 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.634954 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.634958 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.634963 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.634967 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.634971 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.634976 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.634980 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.634987 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.634991 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.634995 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.634999 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.635004 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.635008 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.635012 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.635018 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.635026 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.635032 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.635038 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.635044 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.635050 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:31 2026-03-09 19:10:03.635056 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.635062 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.635069 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.635075 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.635080 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.635098 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.635105 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.635112 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.635118 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.635124 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.635130 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.635138 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.635143 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.635165 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.635170 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.635174 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.635178 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.635183 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.635187 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.635191 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.635198 | controller | \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.635203 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635207 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-09 19:10:03.635212 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-09 19:10:03.635216 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.635220 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.635224 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-09 19:10:03.635229 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-09 19:10:03.635233 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-09 19:10:03.635237 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.635241 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.635246 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-09 19:10:03.635250 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635254 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.635258 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-09 19:10:03.635263 | controller | CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635267 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.635271 | controller | 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.635282 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.635289 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.635294 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.635298 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.635302 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.635307 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.635311 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.635315 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.635369 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.635374 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.635378 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.635383 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.635387 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.635391 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.635397 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.635412 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.635421 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.635427 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.635433 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.635439 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635445 | controller | | ++ base64 --decode\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635451 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.635457 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-09 19:10:03.635463 | controller | CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635468 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.635472 | controller | 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.635480 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.635484 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.635489 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.635496 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.635511 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.635517 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.635526 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.635540 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.635546 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.635552 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.635558 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.635563 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.635569 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.635574 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.635580 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.635586 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.635591 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.635597 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.635603 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.635609 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635615 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635621 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.635626 | controller | 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.635631 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.635635 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.635643 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.635647 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.635652 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.635656 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.635660 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.635667 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.635672 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.635676 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.635680 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.635696 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.635701 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.635705 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.635709 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.635713 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.635718 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.635722 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.635726 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.635732 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 2026-03-09 19:10:03.635737 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.635741 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.635746 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.635750 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.635754 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.635758 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.635764 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.635769 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.635773 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.635778 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.635784 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.635804 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.635814 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.635820 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.635825 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.635831 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.635837 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.635843 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.635849 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.635863 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.635868 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.635872 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635876 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-09 19:10:03.635881 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-09 19:10:03.635885 | controller | CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635889 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.635893 | controller | 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.635898 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.635902 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.635906 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.635910 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.635919 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.635923 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.635927 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.635932 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.635936 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.635940 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.635944 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.635949 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.635953 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.635957 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.635964 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.635968 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.635972 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.635976 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.635981 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.635985 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635989 | controller | | ++ base64 --decode\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.635995 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.636004 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-09 19:10:03.636018 | controller | CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636043 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.636051 | controller | 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.636058 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.636064 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.636070 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.636080 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.636086 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.636092 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.636098 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.636106 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.636113 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.636119 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.636127 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.636134 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.636140 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.636179 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.636188 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.636194 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.636200 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.636206 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.636212 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.636222 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636227 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636231 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.636235 | controller | 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.636239 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.636244 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.636248 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.636252 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.636259 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.636274 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.636279 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.636284 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.636288 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.636292 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.636297 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.636301 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.636305 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.636309 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.636314 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.636318 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.636322 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.636326 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.636331 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.636335 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 2026-03-09 19:10:03.636339 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.636344 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.636348 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.636354 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.636358 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.636363 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.636367 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.636371 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.636377 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.636381 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.636386 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.636390 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.636394 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.636406 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.636410 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.636414 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.636419 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.636423 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.636454 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.636458 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.636462 | controller | \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.636467 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:33 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636471 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-09 19:10:03.636475 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-09 19:10:03.636480 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.636484 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.636491 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-09 19:10:03.636497 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-09 19:10:03.636503 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-09 19:10:03.636509 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.636515 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.636521 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-09 19:10:03.636527 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:33 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636531 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.636538 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-09 19:10:03.636543 | controller | CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636550 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.636556 | controller | 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.636560 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.636565 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.636569 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.636573 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.636580 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.636584 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.636588 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.636592 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.636597 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.636609 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.636614 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.636618 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.636622 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.636627 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.636631 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.636635 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.636639 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.636644 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.636648 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.636652 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636656 | controller | | ++ base64 --decode\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636661 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.636665 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-09 19:10:03.636673 | controller | CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636679 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.636685 | controller | 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.636691 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.636697 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.636703 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.636709 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.636716 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.636721 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.636727 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.636733 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.636743 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.636749 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.636755 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.636760 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.636766 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.636782 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.636788 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.636794 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.636800 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.636806 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.636811 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.636817 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636823 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.636829 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.636841 | controller | 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.636847 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.636854 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.636860 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.636866 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.636872 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.636878 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.636884 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.636890 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.636896 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.636902 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.636908 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.636914 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.636925 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.636931 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.636938 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.636944 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.636951 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.636957 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.636964 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.636970 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 2026-03-09 19:10:03.636988 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.636996 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.637002 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.637012 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.637018 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.637024 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.637028 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.637032 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.637037 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.637041 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.637045 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.637049 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.637054 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.637058 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.637062 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.637066 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.637071 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.637078 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.637082 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.637087 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.637091 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.637096 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637100 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-09 19:10:03.637104 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-09 19:10:03.637109 | controller | CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637113 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.637117 | controller | 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.637121 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.637128 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.637132 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.637164 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.637173 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.637178 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.637182 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.637186 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.637190 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.637195 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.637199 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.637203 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.637207 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.637211 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.637216 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.637220 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.637224 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.637228 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.637233 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.637237 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637241 | controller | | ++ base64 --decode\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637245 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.637250 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-09 19:10:03.637254 | controller | CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637260 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.637266 | controller | 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.637276 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.637285 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.637291 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.637297 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.637302 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.637306 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.637310 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.637322 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.637326 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.637331 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.637337 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.637343 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.637349 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.637355 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.637361 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.637367 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.637373 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.637379 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.637385 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.637391 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637398 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637404 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.637410 | controller | 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.637417 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.637427 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.637434 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.637440 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.637450 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.637457 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.637463 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.637469 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.637475 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.637481 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.637487 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.637493 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.637499 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.637516 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.637523 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.637529 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.637535 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.637540 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.637546 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.637551 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 2026-03-09 19:10:03.637557 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.637562 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.637566 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.637571 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.637575 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.637582 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.637586 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.637590 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.637597 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.637601 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.637606 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.637610 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.637614 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.637618 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.637623 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.637627 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.637631 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.637635 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.637639 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.637644 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.637648 | controller | \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.637652 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637657 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-09 19:10:03.637663 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-09 19:10:03.637677 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.637685 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.637691 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-09 19:10:03.637697 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-09 19:10:03.637702 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-09 19:10:03.637708 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.637714 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-09 19:10:03.637720 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-09 19:10:03.637730 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637736 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.637742 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-09 19:10:03.637748 | controller | CERTIFICATE-----\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637754 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.637759 | controller | 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.637765 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.637771 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.637777 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.637783 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.637789 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.637795 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.637801 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.637808 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.637814 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.637825 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.637832 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.637838 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.637844 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.637849 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.637855 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.637861 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.637867 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.637873 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.637890 | controller | \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.637901 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637907 | controller | | ++ base64 --decode\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637914 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.637920 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-09 19:10:03.637926 | controller | CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.637933 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.637939 | controller | 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.637945 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.637951 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.637957 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.637963 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.637970 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.637977 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.637983 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.637989 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.637995 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.638001 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.638007 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.638013 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.638019 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.638030 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.638037 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.638043 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.638049 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.638055 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.638068 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.638075 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638082 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638088 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.638094 | controller | 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.638112 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.638121 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.638128 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.638134 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.638140 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.638164 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.638176 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.638182 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.638188 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.638194 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.638200 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.638206 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.638211 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.638217 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.638223 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.638229 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.638235 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.638243 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.638247 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.638252 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 2026-03-09 19:10:03.638258 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.638264 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.638270 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n 2026-03-09 19:10:03.638277 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.638283 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n 2026-03-09 19:10:03.638289 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.638295 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n 2026-03-09 19:10:03.638300 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.638306 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.638327 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.638334 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.638340 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.638346 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n 2026-03-09 19:10:03.638352 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.638357 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n 2026-03-09 19:10:03.638363 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.638369 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n 2026-03-09 19:10:03.638375 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.638381 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n 2026-03-09 19:10:03.638385 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.638390 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.638394 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638398 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-09 19:10:03.638403 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-09 19:10:03.638409 | controller | CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638413 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.638418 | controller | 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.638422 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.638426 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.638430 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.638437 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.638441 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.638446 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.638450 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.638454 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.638458 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.638463 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.638469 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.638475 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.638492 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.638499 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.638506 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.638512 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.638518 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.638524 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.638530 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.638536 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638542 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.638548 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.638558 | controller | --decode\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638565 | controller | | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 2026-03-09 19:10:03.638571 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.638578 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.638584 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.638590 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.638596 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.638602 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.638608 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.638619 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.638626 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.638632 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.638638 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.638644 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.638650 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.638656 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.638662 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.638668 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.638674 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.638680 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.638697 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.638703 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.638710 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.638719 | controller | CERTIFICATE-----'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638725 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638734 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.638740 | controller | 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.638746 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.638752 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.638757 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.638763 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.638769 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.638775 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.638781 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.638787 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.638795 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.638801 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.638807 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.638812 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.638818 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.638824 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.638829 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.638833 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.638837 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.638842 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.638846 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.638850 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 2026-03-09 19:10:03.638855 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.638859 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.638863 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.638876 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.638881 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.638885 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.638890 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.638894 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.638898 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.638904 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.638909 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.638913 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.638919 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.638925 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.638931 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.638937 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.638944 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.638950 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.638956 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.638962 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.638968 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.638974 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-09 19:10:03.638980 | controller | | test step completed 1-deploy-with-tlse\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.638986 | controller | | starting test step 2-change-public-svc-certificate\n logger.go:42: 19:02:36 2026-03-09 19:10:03.638992 | controller | | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh 2026-03-09 19:10:03.638996 | controller | -c oc patch Certificate -n $NAMESPACE watcher-public-svc --type='json' -p='[{\"op\": 2026-03-09 19:10:03.639000 | controller | \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-public.watcher-kuttl-default.svc', 2026-03-09 19:10:03.639005 | controller | 'watcher-public.watcher-kuttl-default.svc.cluster.local']}]'\n ]\n logger.go:42: 2026-03-09 19:10:03.639012 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | certificate.cert-manager.io/watcher-public-svc 2026-03-09 19:10:03.639016 | controller | patched\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639020 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-09 19:10:03.639024 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc 2026-03-09 19:10:03.639029 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.639033 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.639037 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-09 19:10:03.639041 | controller | \ ]\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639054 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.639059 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639063 | controller | | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639067 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.639072 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.639076 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639080 | controller | | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n logger.go:42: 2026-03-09 19:10:03.639088 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.639093 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639097 | controller | | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n logger.go:42: 2026-03-09 19:10:03.639101 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.639105 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639110 | controller | | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n logger.go:42: 2026-03-09 19:10:03.639114 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.639118 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639122 | controller | | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 2026-03-09 19:10:03.639127 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.639131 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639135 | controller | | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 2026-03-09 19:10:03.639139 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.639157 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639165 | controller | | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n logger.go:42: 2026-03-09 19:10:03.639171 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.639175 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639181 | controller | | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n logger.go:42: 2026-03-09 19:10:03.639186 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.639190 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639194 | controller | | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n logger.go:42: 2026-03-09 19:10:03.639198 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.639203 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639207 | controller | | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n logger.go:42: 2026-03-09 19:10:03.639211 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.639215 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639219 | controller | | -----END CERTIFICATE-----'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639224 | controller | | ++ base64 --decode\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639228 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.639232 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639245 | controller | | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639249 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.639254 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n 2026-03-09 19:10:03.639258 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639262 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 2026-03-09 19:10:03.639266 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n 2026-03-09 19:10:03.639271 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639275 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 2026-03-09 19:10:03.639279 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n 2026-03-09 19:10:03.639283 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639288 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 2026-03-09 19:10:03.639292 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n 2026-03-09 19:10:03.639296 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639300 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 2026-03-09 19:10:03.639304 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.639311 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639315 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 2026-03-09 19:10:03.639319 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n 2026-03-09 19:10:03.639324 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639328 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 2026-03-09 19:10:03.639332 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n 2026-03-09 19:10:03.639336 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639341 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 2026-03-09 19:10:03.639345 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n 2026-03-09 19:10:03.639349 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639353 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 2026-03-09 19:10:03.639357 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n 2026-03-09 19:10:03.639362 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639366 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 2026-03-09 19:10:03.639370 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n 2026-03-09 19:10:03.639374 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639379 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 2026-03-09 19:10:03.639383 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n 2026-03-09 19:10:03.639387 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639391 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639396 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.639400 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n 2026-03-09 19:10:03.639404 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639416 | controller | | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n logger.go:42: 2026-03-09 19:10:03.639420 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n 2026-03-09 19:10:03.639425 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639429 | controller | | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n logger.go:42: 2026-03-09 19:10:03.639435 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n 2026-03-09 19:10:03.639440 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639446 | controller | | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n logger.go:42: 2026-03-09 19:10:03.639450 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n 2026-03-09 19:10:03.639454 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639459 | controller | | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 2026-03-09 19:10:03.639463 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.639467 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639471 | controller | | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 2026-03-09 19:10:03.639476 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n 2026-03-09 19:10:03.639480 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639484 | controller | | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n logger.go:42: 2026-03-09 19:10:03.639488 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n 2026-03-09 19:10:03.639492 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639497 | controller | | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n logger.go:42: 2026-03-09 19:10:03.639501 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n 2026-03-09 19:10:03.639505 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639509 | controller | | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n logger.go:42: 2026-03-09 19:10:03.639513 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n 2026-03-09 19:10:03.639518 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639522 | controller | | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n logger.go:42: 2026-03-09 19:10:03.639526 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n 2026-03-09 19:10:03.639530 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639535 | controller | | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 2026-03-09 19:10:03.639539 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.639543 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639547 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n logger.go:42: 2026-03-09 19:10:03.639552 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n 2026-03-09 19:10:03.639556 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639560 | controller | | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n logger.go:42: 2026-03-09 19:10:03.639564 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n 2026-03-09 19:10:03.639568 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639575 | controller | | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n logger.go:42: 2026-03-09 19:10:03.639579 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n 2026-03-09 19:10:03.639589 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639593 | controller | | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n logger.go:42: 2026-03-09 19:10:03.639597 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n 2026-03-09 19:10:03.639602 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639606 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n logger.go:42: 2026-03-09 19:10:03.639610 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n 2026-03-09 19:10:03.639614 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639619 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n logger.go:42: 2026-03-09 19:10:03.639623 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n 2026-03-09 19:10:03.639629 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639633 | controller | | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n logger.go:42: 2026-03-09 19:10:03.639637 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n 2026-03-09 19:10:03.639641 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639646 | controller | | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n logger.go:42: 2026-03-09 19:10:03.639650 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n 2026-03-09 19:10:03.639654 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639658 | controller | | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n logger.go:42: 2026-03-09 19:10:03.639662 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n 2026-03-09 19:10:03.639667 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639671 | controller | | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n logger.go:42: 2026-03-09 19:10:03.639675 | controller | 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n 2026-03-09 19:10:03.639679 | controller | \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639684 | controller | | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639688 | controller | | + exit 1\n logger.go:42: 19:02:37 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639692 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-09 19:10:03.639698 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc 2026-03-09 19:10:03.639702 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.639706 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.639712 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-09 19:10:03.639716 | controller | \ ]\n logger.go:42: 19:02:37 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639721 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.639725 | controller | \ logger.go:42: 19:02:38 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639729 | controller | | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-09 19:10:03.639733 | controller | 19:02:38 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=\n 2026-03-09 19:10:03.639737 | controller | \ logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639742 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-09 19:10:03.639746 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc 2026-03-09 19:10:03.639750 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.639754 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.639763 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-09 19:10:03.639768 | controller | \ ]\n logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639772 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.639776 | controller | \ logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639802 | controller | | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-09 19:10:03.639808 | controller | 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=\n 2026-03-09 19:10:03.639813 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639817 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-09 19:10:03.639822 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc 2026-03-09 19:10:03.639826 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.639830 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.639834 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-09 19:10:03.639839 | controller | \ ]\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639843 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-09 19:10:03.639847 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639851 | controller | | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639856 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.639860 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n 2026-03-09 19:10:03.639864 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639868 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 2026-03-09 19:10:03.639872 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n 2026-03-09 19:10:03.639879 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639883 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 2026-03-09 19:10:03.639887 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n 2026-03-09 19:10:03.639892 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639898 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 2026-03-09 19:10:03.639904 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n 2026-03-09 19:10:03.639909 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639913 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 2026-03-09 19:10:03.639918 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.639922 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639926 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 2026-03-09 19:10:03.639930 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n 2026-03-09 19:10:03.639934 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639939 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 2026-03-09 19:10:03.639943 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n 2026-03-09 19:10:03.639947 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639951 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 2026-03-09 19:10:03.639956 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n 2026-03-09 19:10:03.639966 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639970 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 2026-03-09 19:10:03.639974 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n 2026-03-09 19:10:03.639979 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639983 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 2026-03-09 19:10:03.639987 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n 2026-03-09 19:10:03.639991 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.639996 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 2026-03-09 19:10:03.640000 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n 2026-03-09 19:10:03.640004 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640008 | controller | | ++ base64 --decode\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640013 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.640019 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640023 | controller | | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640027 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.640032 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n 2026-03-09 19:10:03.640036 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640040 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 2026-03-09 19:10:03.640044 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n 2026-03-09 19:10:03.640049 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640053 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 2026-03-09 19:10:03.640057 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n 2026-03-09 19:10:03.640061 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640065 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 2026-03-09 19:10:03.640070 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n 2026-03-09 19:10:03.640074 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640078 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 2026-03-09 19:10:03.640082 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.640087 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640091 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 2026-03-09 19:10:03.640095 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n 2026-03-09 19:10:03.640099 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640104 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 2026-03-09 19:10:03.640108 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n 2026-03-09 19:10:03.640112 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640119 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 2026-03-09 19:10:03.640124 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n 2026-03-09 19:10:03.640128 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640138 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 2026-03-09 19:10:03.640142 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n 2026-03-09 19:10:03.640161 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640169 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 2026-03-09 19:10:03.640173 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n 2026-03-09 19:10:03.640177 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640181 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 2026-03-09 19:10:03.640186 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n 2026-03-09 19:10:03.640190 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640194 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640198 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.640203 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n 2026-03-09 19:10:03.640207 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640211 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 2026-03-09 19:10:03.640215 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n 2026-03-09 19:10:03.640220 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640224 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 2026-03-09 19:10:03.640228 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n 2026-03-09 19:10:03.640232 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640236 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 2026-03-09 19:10:03.640241 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n 2026-03-09 19:10:03.640245 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640249 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 2026-03-09 19:10:03.640263 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.640268 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640272 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 2026-03-09 19:10:03.640276 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n 2026-03-09 19:10:03.640280 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640285 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 2026-03-09 19:10:03.640289 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n 2026-03-09 19:10:03.640293 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640297 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 2026-03-09 19:10:03.640302 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n 2026-03-09 19:10:03.640308 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640312 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 2026-03-09 19:10:03.640317 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n 2026-03-09 19:10:03.640321 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640325 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 2026-03-09 19:10:03.640335 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n 2026-03-09 19:10:03.640340 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640344 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 2026-03-09 19:10:03.640349 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----' 2026-03-09 19:10:03.640353 | controller | '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640357 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.640361 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n 2026-03-09 19:10:03.640366 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640372 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 2026-03-09 19:10:03.640376 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n 2026-03-09 19:10:03.640380 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640384 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 2026-03-09 19:10:03.640389 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n 2026-03-09 19:10:03.640393 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640397 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 2026-03-09 19:10:03.640401 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n 2026-03-09 19:10:03.640405 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640410 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 2026-03-09 19:10:03.640414 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n 2026-03-09 19:10:03.640418 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640422 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 2026-03-09 19:10:03.640427 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n 2026-03-09 19:10:03.640431 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640435 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 2026-03-09 19:10:03.640439 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n 2026-03-09 19:10:03.640445 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640450 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 2026-03-09 19:10:03.640454 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n 2026-03-09 19:10:03.640458 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640462 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 2026-03-09 19:10:03.640467 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n 2026-03-09 19:10:03.640471 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640475 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 2026-03-09 19:10:03.640479 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n 2026-03-09 19:10:03.640484 | controller | \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640488 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 2026-03-09 19:10:03.640492 | controller | 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----' 2026-03-09 19:10:03.640496 | controller | ']'\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.640506 | controller | | test step completed 2-change-public-svc-certificate\n logger.go:42: 19:02:40 2026-03-09 19:10:03.640510 | controller | | watcher-tls-certs-change/3-change-internal-svc-certificate | starting test step 2026-03-09 19:10:03.640515 | controller | 3-change-internal-svc-certificate\n logger.go:42: 19:02:40 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640519 | controller | | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc 2026-03-09 19:10:03.640523 | controller | --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-internal.watcher-kuttl-default.svc', 2026-03-09 19:10:03.640529 | controller | 'watcher-internal.watcher-kuttl-default.svc.cluster.local']}]'\n ]\n logger.go:42: 2026-03-09 19:10:03.640534 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | certificate.cert-manager.io/watcher-internal-svc 2026-03-09 19:10:03.640539 | controller | patched\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640543 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-09 19:10:03.640547 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n secret_cert=$(oc 2026-03-09 19:10:03.640552 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-09 19:10:03.640556 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-09 19:10:03.640560 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-09 19:10:03.640564 | controller | \ ]\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640569 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-09 19:10:03.640573 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640577 | controller | | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640581 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.640587 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.640591 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640596 | controller | | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n logger.go:42: 2026-03-09 19:10:03.640600 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.640604 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640609 | controller | | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n logger.go:42: 2026-03-09 19:10:03.640613 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.640620 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640625 | controller | | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n logger.go:42: 2026-03-09 19:10:03.640632 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.640638 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640644 | controller | | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 2026-03-09 19:10:03.640650 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.640656 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640662 | controller | | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 2026-03-09 19:10:03.640667 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.640673 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640679 | controller | | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n logger.go:42: 2026-03-09 19:10:03.640685 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.640691 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640707 | controller | | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n logger.go:42: 2026-03-09 19:10:03.640715 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.640721 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640727 | controller | | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n logger.go:42: 2026-03-09 19:10:03.640734 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.640741 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640747 | controller | | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n logger.go:42: 2026-03-09 19:10:03.640753 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.640758 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640768 | controller | | -----END CERTIFICATE-----'\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640775 | controller | | ++ base64 --decode\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640781 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-09 19:10:03.640787 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640793 | controller | | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640799 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.640805 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.640811 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640817 | controller | | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n logger.go:42: 2026-03-09 19:10:03.640823 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.640829 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640835 | controller | | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n logger.go:42: 2026-03-09 19:10:03.640841 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.640847 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640853 | controller | | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n logger.go:42: 2026-03-09 19:10:03.640859 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.640868 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640873 | controller | | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 2026-03-09 19:10:03.640877 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.640881 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640885 | controller | | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 2026-03-09 19:10:03.640890 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.640894 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640898 | controller | | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n logger.go:42: 2026-03-09 19:10:03.640902 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.640907 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640911 | controller | | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n logger.go:42: 2026-03-09 19:10:03.640927 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.640932 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640938 | controller | | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n logger.go:42: 2026-03-09 19:10:03.640944 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.640950 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640956 | controller | | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n logger.go:42: 2026-03-09 19:10:03.640963 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.640969 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640975 | controller | | -----END CERTIFICATE-----'\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640981 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.640987 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-09 19:10:03.640992 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n 2026-03-09 19:10:03.640997 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641001 | controller | | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n logger.go:42: 2026-03-09 19:10:03.641005 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n 2026-03-09 19:10:03.641010 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641014 | controller | | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n logger.go:42: 2026-03-09 19:10:03.641018 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n 2026-03-09 19:10:03.641022 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641027 | controller | | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n logger.go:42: 2026-03-09 19:10:03.641031 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n 2026-03-09 19:10:03.641035 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641039 | controller | | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 2026-03-09 19:10:03.641044 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n 2026-03-09 19:10:03.641048 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641052 | controller | | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 2026-03-09 19:10:03.641057 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n 2026-03-09 19:10:03.641061 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641067 | controller | | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n logger.go:42: 2026-03-09 19:10:03.641073 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n 2026-03-09 19:10:03.641079 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641088 | controller | | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n logger.go:42: 2026-03-09 19:10:03.641095 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n 2026-03-09 19:10:03.641101 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641107 | controller | | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n logger.go:42: 2026-03-09 19:10:03.641112 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n 2026-03-09 19:10:03.641118 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641123 | controller | | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n logger.go:42: 2026-03-09 19:10:03.641139 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n 2026-03-09 19:10:03.641165 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641179 | controller | | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 2026-03-09 19:10:03.641185 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n 2026-03-09 19:10:03.641191 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641197 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n logger.go:42: 2026-03-09 19:10:03.641203 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n 2026-03-09 19:10:03.641209 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641215 | controller | | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n logger.go:42: 2026-03-09 19:10:03.641221 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n 2026-03-09 19:10:03.641227 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641238 | controller | | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n logger.go:42: 2026-03-09 19:10:03.641244 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n 2026-03-09 19:10:03.641248 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641252 | controller | | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n logger.go:42: 2026-03-09 19:10:03.641257 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-09 19:10:03.641261 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641266 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n logger.go:42: 2026-03-09 19:10:03.641272 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-09 19:10:03.641279 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641285 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n logger.go:42: 2026-03-09 19:10:03.641291 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n 2026-03-09 19:10:03.641300 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641309 | controller | | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n logger.go:42: 2026-03-09 19:10:03.641315 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n 2026-03-09 19:10:03.641321 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641327 | controller | | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n logger.go:42: 2026-03-09 19:10:03.641333 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n 2026-03-09 19:10:03.641338 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641344 | controller | | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n logger.go:42: 2026-03-09 19:10:03.641350 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n 2026-03-09 19:10:03.641356 | controller | \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641362 | controller | | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n logger.go:42: 2026-03-09 19:10:03.641368 | controller | 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END 2026-03-09 19:10:03.641373 | controller | CERTIFICATE-----' ']'\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.641379 | controller | | test step completed 3-change-internal-svc-certificate\n logger.go:42: 19:02:41 2026-03-09 19:10:03.641385 | controller | | watcher-tls-certs-change/4-cleanup-watcher | starting test step 4-cleanup-watcher\n 2026-03-09 19:10:03.641390 | controller | \ logger.go:42: 19:02:47 | watcher-tls-certs-change/4-cleanup-watcher | test step 2026-03-09 19:10:03.641404 | controller | completed 4-cleanup-watcher\n logger.go:42: 19:02:47 | watcher-tls-certs-change/5-clenaup-certs 2026-03-09 19:10:03.641408 | controller | | starting test step 5-clenaup-certs\n logger.go:42: 19:02:47 | watcher-tls-certs-change/5-clenaup-certs 2026-03-09 19:10:03.641413 | controller | | test step completed 5-clenaup-certs\n logger.go:42: 19:02:47 | watcher-tls-certs-change 2026-03-09 19:10:03.641417 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-tls\n logger.go:42: 2026-03-09 19:10:03.641421 | controller | 19:02:47 | watcher-tls | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-09 19:10:03.641426 | controller | \ logger.go:42: 19:02:47 | watcher-tls/0-cleanup-watcher | starting test step 2026-03-09 19:10:03.641430 | controller | 0-cleanup-watcher\n logger.go:42: 19:02:47 | watcher-tls/0-cleanup-watcher | 2026-03-09 19:10:03.641434 | controller | test step completed 0-cleanup-watcher\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641438 | controller | | starting test step 1-deploy-with-tlse\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641443 | controller | | Certificate:watcher-kuttl-default/watcher-internal-svc created\n logger.go:42: 2026-03-09 19:10:03.641447 | controller | 19:02:47 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc 2026-03-09 19:10:03.641451 | controller | created\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.641455 | controller | created\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-09 19:10:03.641459 | controller | [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-09 19:10:03.641464 | controller | service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc 2026-03-09 19:10:03.641468 | controller | exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.641474 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} 2026-03-09 19:10:03.641479 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-09 19:10:03.641483 | controller | -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-09 19:10:03.641487 | controller | ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o 2026-03-09 19:10:03.641491 | controller | jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-09 19:10:03.641496 | controller | get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-09 19:10:03.641500 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-09 19:10:03.641506 | controller | ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.641511 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-09 19:10:03.641515 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.641520 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.641524 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.641528 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.641532 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.641538 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.641542 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.641547 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.641551 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.641555 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641559 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.641564 | controller | value -c Name -c Type\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641568 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:02:49 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641572 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641576 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-09 19:10:03.641580 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-09 19:10:03.641585 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.641589 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-09 19:10:03.641599 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.641604 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-09 19:10:03.641608 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret 2026-03-09 19:10:03.641612 | controller | watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" 2026-03-09 19:10:03.641617 | controller | == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.641621 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-09 19:10:03.641627 | controller | == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.641631 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-09 19:10:03.641636 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.641640 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.641644 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.641648 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.641653 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.641657 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.641661 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.641665 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.641669 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.641674 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641678 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.641682 | controller | value -c Name -c Type\n logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641686 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:02:53 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641690 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641695 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-09 19:10:03.641699 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-09 19:10:03.641705 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.641709 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-09 19:10:03.641713 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.641717 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-09 19:10:03.641722 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret 2026-03-09 19:10:03.641726 | controller | watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" 2026-03-09 19:10:03.641730 | controller | == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.641734 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-09 19:10:03.641740 | controller | == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.641745 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-09 19:10:03.641749 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.641753 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.641757 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.641761 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.641766 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.641770 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.641776 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.641780 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.641784 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.641795 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641799 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641804 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.641808 | controller | value -c Name -c Type\n logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641812 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641817 | controller | | ++ grep watcher\n logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | 2026-03-09 19:10:03.641821 | controller | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.641825 | controller | value -c Name -c Type -c ID\n logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641829 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641834 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641838 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.641842 | controller | \ logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.641846 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641851 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.641855 | controller | \ logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + '[' -n '' ']'\n 2026-03-09 19:10:03.641859 | controller | \ logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | running command: [sh 2026-03-09 19:10:03.641863 | controller | -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-09 19:10:03.641868 | controller | service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc 2026-03-09 19:10:03.641872 | controller | exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.641876 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} 2026-03-09 19:10:03.641880 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-09 19:10:03.641884 | controller | -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-09 19:10:03.641889 | controller | ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o 2026-03-09 19:10:03.641893 | controller | jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-09 19:10:03.641897 | controller | get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-09 19:10:03.641901 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-09 19:10:03.641907 | controller | ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.641912 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-09 19:10:03.641916 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.641920 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.641926 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.641930 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.641935 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.641939 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.641943 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.641947 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.641951 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.641956 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641960 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.641964 | controller | value -c Name -c Type\n logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641968 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641972 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641977 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.641981 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.641991 | controller | | ++ grep watcher\n logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | 2026-03-09 19:10:03.641995 | controller | ++ awk '{print $1}'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642000 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642004 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.642008 | controller | \ logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.642013 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642017 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.642021 | controller | \ logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.642029 | controller | ']'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n 2026-03-09 19:10:03.642033 | controller | \ logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.642037 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-09 19:10:03.642041 | controller | 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n logger.go:42: 2026-03-09 19:10:03.642046 | controller | 19:03:04 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.642050 | controller | 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 19:03:04 2026-03-09 19:10:03.642054 | controller | | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n 2026-03-09 19:10:03.642058 | controller | \ logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.642063 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-09 19:10:03.642067 | controller | 19:03:04 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-09 19:10:03.642073 | controller | 19:03:04 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 2026-03-09 19:10:03.642077 | controller | 19:03:04 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.642082 | controller | -- openstack endpoint list\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642086 | controller | | ++ grep -c https\n logger.go:42: 19:03:06 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642090 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642094 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-09 19:10:03.642099 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-09 19:10:03.642103 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.642107 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-09 19:10:03.642111 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.642116 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-09 19:10:03.642120 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret 2026-03-09 19:10:03.642124 | controller | watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" 2026-03-09 19:10:03.642130 | controller | == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.642134 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-09 19:10:03.642139 | controller | == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.642162 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-09 19:10:03.642173 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.642180 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.642186 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.642192 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.642198 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.642207 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.642213 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.642225 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.642230 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.642234 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642238 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.642243 | controller | value -c Name -c Type\n logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642247 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642251 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642255 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642259 | controller | | ++ grep watcher\n logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | 2026-03-09 19:10:03.642264 | controller | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.642272 | controller | value -c Name -c Type -c ID\n logger.go:42: 19:03:11 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642276 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642280 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.642285 | controller | \ logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.642289 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642293 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.642297 | controller | \ logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.642302 | controller | ']'\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n 2026-03-09 19:10:03.642306 | controller | \ logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 2026-03-09 19:10:03.642310 | controller | 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret 2026-03-09 19:10:03.642314 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-09 19:10:03.642319 | controller | 19:03:12 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.642323 | controller | 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n 2026-03-09 19:10:03.642327 | controller | \ logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.642331 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-09 19:10:03.642336 | controller | 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 19:03:12 2026-03-09 19:10:03.642340 | controller | | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 19:03:12 2026-03-09 19:10:03.642344 | controller | | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 19:03:12 2026-03-09 19:10:03.642348 | controller | | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.642352 | controller | -- openstack endpoint list\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642357 | controller | | ++ grep -c https\n logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642361 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642365 | controller | | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-09 19:10:03.642369 | controller | \ logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n 2026-03-09 19:10:03.642374 | controller | \ logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + exit 0\n logger.go:42: 2026-03-09 19:10:03.642378 | controller | 19:03:16 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.642382 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.642386 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.642393 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep 2026-03-09 19:10:03.642397 | controller | watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher 2026-03-09 19:10:03.642401 | controller | -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} 2026-03-09 19:10:03.642406 | controller | watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get 2026-03-09 19:10:03.642418 | controller | -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 2026-03-09 19:10:03.642422 | controller | -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.642427 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-09 19:10:03.642431 | controller | == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.642435 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-09 19:10:03.642439 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.642444 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.642448 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.642452 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.642456 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.642461 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.642465 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.642469 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.642473 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.642477 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642482 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.642486 | controller | value -c Name -c Type\n logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642490 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642494 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642498 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.642503 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642507 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642511 | controller | | ++ grep watcher\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | 2026-03-09 19:10:03.642515 | controller | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642520 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.642524 | controller | \ logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.642528 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642532 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.642538 | controller | \ logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.642543 | controller | ']'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n 2026-03-09 19:10:03.642547 | controller | \ logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 2026-03-09 19:10:03.642553 | controller | 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret 2026-03-09 19:10:03.642557 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-09 19:10:03.642561 | controller | 19:03:20 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.642566 | controller | 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 19:03:20 2026-03-09 19:10:03.642570 | controller | | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.642574 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642578 | controller | | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 2026-03-09 19:10:03.642583 | controller | 19:03:20 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-09 19:10:03.642587 | controller | 19:03:20 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 2026-03-09 19:10:03.642591 | controller | 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c https\n logger.go:42: 2026-03-09 19:10:03.642602 | controller | 19:03:20 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.642606 | controller | -- openstack endpoint list\n logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642611 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse 2026-03-09 19:10:03.642615 | controller | | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-09 19:10:03.642619 | controller | \ logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n 2026-03-09 19:10:03.642623 | controller | \ logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + exit 0\n logger.go:42: 2026-03-09 19:10:03.642628 | controller | 19:03:22 | watcher-tls/1-deploy-with-tlse | test step completed 1-deploy-with-tlse\n 2026-03-09 19:10:03.642632 | controller | \ logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls\n 2026-03-09 19:10:03.642636 | controller | \ logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-09 19:10:03.642642 | controller | set -euxo pipefail\n oc patch oscp -n $NAMESPACE openstack --type='json' 2026-03-09 19:10:03.642646 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\", 2026-03-09 19:10:03.642651 | controller | \"value\": \"Request\"}]'\n ]\n logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.642655 | controller | | + oc patch oscp -n watcher-kuttl-default openstack --type=json '-p=[{\"op\": \"replace\", 2026-03-09 19:10:03.642659 | controller | \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\", \"value\": 2026-03-09 19:10:03.642664 | controller | \"Request\"}]'\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack 2026-03-09 19:10:03.642668 | controller | patched\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | running command: 2026-03-09 19:10:03.642672 | controller | [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n # 2026-03-09 19:10:03.642676 | controller | Get pod names for each watcher service\n APIPOD=$(oc get pods -l service=watcher-api 2026-03-09 19:10:03.642681 | controller | -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc get pods -l service=watcher-applier 2026-03-09 19:10:03.642685 | controller | -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc get pods 2026-03-09 19:10:03.642689 | controller | -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n \n 2026-03-09 19:10:03.642693 | controller | \ # Verify memcached mTLS config parameters in watcher-api config\n if 2026-03-09 19:10:03.642698 | controller | [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n 2026-03-09 19:10:03.642702 | controller | \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642708 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.642712 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642717 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.642721 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642725 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.642729 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642734 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify 2026-03-09 19:10:03.642738 | controller | mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh 2026-03-09 19:10:03.642742 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.642746 | controller | \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.642751 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.642755 | controller | \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.642759 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.642763 | controller | \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 2026-03-09 19:10:03.642768 | controller | 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier 2026-03-09 19:10:03.642772 | controller | config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier 2026-03-09 19:10:03.642776 | controller | config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642780 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.642785 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642796 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.642800 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642805 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.642809 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642813 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.642818 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642822 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.642826 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642830 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.642835 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642840 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.642845 | controller | \ exit 1\n fi\n \n # Verify memcached mTLS config parameters 2026-03-09 19:10:03.642849 | controller | in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n 2026-03-09 19:10:03.642853 | controller | \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo 2026-03-09 19:10:03.642861 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642868 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.642874 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642880 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.642886 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642892 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.642898 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642903 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.642908 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642913 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.642917 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642921 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.642925 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.642930 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.642934 | controller | \ exit 1\n fi\n ]\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.642938 | controller | | + oc project watcher-kuttl-default\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.642942 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.642947 | controller | \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.642951 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.642955 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.642959 | controller | | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n 2026-03-09 19:10:03.642964 | controller | \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n 2026-03-09 19:10:03.642968 | controller | \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.642972 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.642976 | controller | | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:23 2026-03-09 19:10:03.642983 | controller | | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 2026-03-09 19:10:03.642987 | controller | 19:03:23 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n 2026-03-09 19:10:03.642992 | controller | \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n 2026-03-09 19:10:03.642996 | controller | \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.643007 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643012 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.643016 | controller | \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.643023 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.643027 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.643033 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.643037 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.643042 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.643046 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.643050 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.643054 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.643059 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.643063 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.643067 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.643071 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.643076 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.643080 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.643084 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.643088 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.643093 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.643097 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.643101 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-09 19:10:03.643105 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.643109 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.643114 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.643118 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.643122 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.643126 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.643131 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.643135 | controller | period = 900\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + '[' 0 == 2026-03-09 19:10:03.643139 | controller | 1 ']'\n logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | running command: 2026-03-09 19:10:03.643157 | controller | [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n # 2026-03-09 19:10:03.643165 | controller | Get pod names for each watcher service\n APIPOD=$(oc get pods -l service=watcher-api 2026-03-09 19:10:03.643172 | controller | -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc get pods -l service=watcher-applier 2026-03-09 19:10:03.643177 | controller | -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc get pods 2026-03-09 19:10:03.643181 | controller | -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n \n 2026-03-09 19:10:03.643185 | controller | \ # Verify memcached mTLS config parameters in watcher-api config\n if 2026-03-09 19:10:03.643189 | controller | [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n 2026-03-09 19:10:03.643193 | controller | \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643204 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.643208 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643213 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.643217 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643221 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.643225 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643230 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify 2026-03-09 19:10:03.643234 | controller | mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643238 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643242 | controller | \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643246 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643251 | controller | \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643255 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643259 | controller | \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 2026-03-09 19:10:03.643263 | controller | 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier 2026-03-09 19:10:03.643268 | controller | config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier 2026-03-09 19:10:03.643272 | controller | config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643276 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.643280 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643284 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.643289 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643293 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.643297 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643301 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.643306 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643310 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.643314 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643318 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.643326 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643330 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.643334 | controller | \ exit 1\n fi\n \n # Verify memcached mTLS config parameters 2026-03-09 19:10:03.643339 | controller | in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n 2026-03-09 19:10:03.643343 | controller | \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo 2026-03-09 19:10:03.643347 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643353 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.643357 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643362 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.643366 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643370 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.643374 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643379 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.643383 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643387 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.643396 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643401 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.643405 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643410 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.643414 | controller | \ exit 1\n fi\n ]\n logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643418 | controller | | + oc project watcher-kuttl-default\n logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643422 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.643426 | controller | \ logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.643431 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643435 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643439 | controller | | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n 2026-03-09 19:10:03.643443 | controller | \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n 2026-03-09 19:10:03.643448 | controller | \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.643452 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643456 | controller | | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:25 2026-03-09 19:10:03.643460 | controller | | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 2026-03-09 19:10:03.643464 | controller | 19:03:25 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n 2026-03-09 19:10:03.643471 | controller | \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n 2026-03-09 19:10:03.643475 | controller | \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.643479 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643485 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.643489 | controller | \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | error: cannot exec into 2026-03-09 19:10:03.643494 | controller | a container in a completed pod; current phase is Failed\n logger.go:42: 19:03:25 2026-03-09 19:10:03.643498 | controller | | watcher-tls/2-patch-mtls | ++ echo\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643502 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | running 2026-03-09 19:10:03.643506 | controller | command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n 2026-03-09 19:10:03.643511 | controller | \ # Get pod names for each watcher service\n APIPOD=$(oc get pods -l 2026-03-09 19:10:03.643515 | controller | service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc 2026-03-09 19:10:03.643519 | controller | get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc 2026-03-09 19:10:03.643523 | controller | get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n 2026-03-09 19:10:03.643527 | controller | \ \n # Verify memcached mTLS config parameters in watcher-api config\n 2026-03-09 19:10:03.643532 | controller | \ if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n 2026-03-09 19:10:03.643536 | controller | \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643540 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.643544 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643549 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.643553 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643557 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.643561 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643566 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify 2026-03-09 19:10:03.643570 | controller | mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643574 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643580 | controller | \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643589 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643593 | controller | \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643598 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643602 | controller | \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 2026-03-09 19:10:03.643606 | controller | 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier 2026-03-09 19:10:03.643610 | controller | config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier 2026-03-09 19:10:03.643615 | controller | config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643621 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.643625 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643629 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.643633 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643638 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.643642 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643646 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.643650 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643654 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.643659 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643663 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.643667 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643671 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.643675 | controller | \ exit 1\n fi\n \n # Verify memcached mTLS config parameters 2026-03-09 19:10:03.643680 | controller | in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n 2026-03-09 19:10:03.643684 | controller | \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo 2026-03-09 19:10:03.643688 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643692 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.643697 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643701 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.643705 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643709 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.643714 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643718 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.643722 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643726 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.643730 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643735 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.643739 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643743 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.643747 | controller | \ exit 1\n fi\n ]\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643752 | controller | | + oc project watcher-kuttl-default\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643758 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.643762 | controller | \ logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.643766 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643775 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643798 | controller | | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n 2026-03-09 19:10:03.643804 | controller | \ logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n 2026-03-09 19:10:03.643809 | controller | \ logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.643813 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643817 | controller | | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:27 2026-03-09 19:10:03.643822 | controller | | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 2026-03-09 19:10:03.643826 | controller | 19:03:27 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n 2026-03-09 19:10:03.643830 | controller | \ logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n 2026-03-09 19:10:03.643834 | controller | \ logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.643839 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643843 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.643849 | controller | \ logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | error: unable to upgrade 2026-03-09 19:10:03.643854 | controller | connection: container not found (\"watcher-api\")\n logger.go:42: 19:03:27 | 2026-03-09 19:10:03.643858 | controller | watcher-tls/2-patch-mtls | ++ echo\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.643862 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | running 2026-03-09 19:10:03.643866 | controller | command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n 2026-03-09 19:10:03.643871 | controller | \ # Get pod names for each watcher service\n APIPOD=$(oc get pods -l 2026-03-09 19:10:03.643875 | controller | service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc 2026-03-09 19:10:03.643879 | controller | get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc 2026-03-09 19:10:03.643883 | controller | get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n 2026-03-09 19:10:03.643888 | controller | \ \n # Verify memcached mTLS config parameters in watcher-api config\n 2026-03-09 19:10:03.643892 | controller | \ if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n 2026-03-09 19:10:03.643896 | controller | \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643900 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.643905 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643909 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.643913 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643917 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.643922 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643929 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify 2026-03-09 19:10:03.643933 | controller | mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643937 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643941 | controller | \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643946 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643950 | controller | \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.643954 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.643958 | controller | \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 2026-03-09 19:10:03.643963 | controller | 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier 2026-03-09 19:10:03.643967 | controller | config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier 2026-03-09 19:10:03.643971 | controller | config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643976 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.643981 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.643992 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.643996 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644000 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.644005 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644009 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.644013 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644019 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.644025 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644041 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.644049 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644059 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.644065 | controller | \ exit 1\n fi\n \n # Verify memcached mTLS config parameters 2026-03-09 19:10:03.644070 | controller | in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n 2026-03-09 19:10:03.644076 | controller | \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo 2026-03-09 19:10:03.644081 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644087 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.644093 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644099 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.644105 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644109 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.644118 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644122 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.644126 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644130 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.644134 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644139 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.644161 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.644168 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.644173 | controller | \ exit 1\n fi\n ]\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.644177 | controller | | + oc project watcher-kuttl-default\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.644181 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.644185 | controller | \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.644189 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.644194 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.644198 | controller | | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n 2026-03-09 19:10:03.644202 | controller | \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n 2026-03-09 19:10:03.644206 | controller | \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.644210 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.644215 | controller | | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:28 2026-03-09 19:10:03.644219 | controller | | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 2026-03-09 19:10:03.644223 | controller | 19:03:28 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n 2026-03-09 19:10:03.644227 | controller | \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n 2026-03-09 19:10:03.644231 | controller | \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.644246 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.644251 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.644255 | controller | \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.644259 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.644264 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.644268 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.644272 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.644276 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.644280 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.644287 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644291 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.644295 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.644299 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.644304 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.644308 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.644312 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.644316 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.644320 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.644325 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.644329 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.644333 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.644337 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644341 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.644345 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.644350 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.644354 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.644358 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.644362 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.644366 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.644373 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.644377 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.644381 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.644385 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.644390 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.644394 | controller | period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.644400 | controller | 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile 2026-03-09 19:10:03.644404 | controller | = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.644408 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.644412 | controller | \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.644416 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.644431 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.644436 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.644440 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.644444 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.644449 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.644453 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644457 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.644461 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.644466 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.644470 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.644474 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.644478 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.644482 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.644487 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.644491 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.644495 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.644499 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.644503 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644507 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.644512 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.644516 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.644520 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.644524 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.644528 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.644533 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.644537 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.644541 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.644545 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.644549 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.644553 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.644558 | controller | period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.644562 | controller | 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile 2026-03-09 19:10:03.644568 | controller | = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.644572 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.644576 | controller | \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.644580 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.644585 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.644589 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.644593 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.644603 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.644608 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.644612 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644616 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.644621 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.644625 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.644629 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.644634 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.644638 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.644642 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.644646 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.644650 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.644655 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.644660 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.644665 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644669 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.644675 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.644679 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.644683 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.644687 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.644692 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.644696 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.644700 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.644704 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.644709 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.644715 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.644719 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.644723 | controller | period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.644727 | controller | 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled 2026-03-09 19:10:03.644732 | controller | = true'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.644736 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.644740 | controller | 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.644744 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.644748 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.644754 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.644759 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.644763 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.644767 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.644771 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644775 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.644780 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.644791 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.644796 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.644800 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.644804 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.644809 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.644813 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.644817 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.644821 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.644825 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.644830 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644834 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.644838 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.644842 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.644846 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.644851 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.644859 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.644875 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.644884 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.644895 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.644901 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.644908 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.644914 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.644920 | controller | period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.644926 | controller | 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n 2026-03-09 19:10:03.644932 | controller | \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.644938 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.644944 | controller | 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.644950 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.644957 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.644962 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.644969 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.644975 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.644981 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.644987 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.644993 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.644999 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.645009 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.645016 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.645022 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.645028 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.645035 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.645054 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.645062 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.645068 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.645074 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.645080 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645086 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.645096 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.645100 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.645104 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.645108 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.645113 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645117 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.645121 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.645125 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645129 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.645134 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.645138 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.645142 | controller | period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.645170 | controller | 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n 2026-03-09 19:10:03.645174 | controller | \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.645179 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.645183 | controller | 19:03:30 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.645187 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.645191 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.645196 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.645200 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.645204 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.645208 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.645213 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645217 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.645221 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.645225 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.645229 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.645234 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.645238 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.645242 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.645246 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.645253 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.645257 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.645261 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.645265 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645278 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.645285 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.645290 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.645294 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.645299 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.645303 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645307 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.645311 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.645317 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645323 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.645330 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.645336 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.645342 | controller | period = 900\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.645348 | controller | 1 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n 2026-03-09 19:10:03.645354 | controller | \ logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.645359 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.645363 | controller | 19:03:30 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.645367 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.645372 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.645376 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.645380 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.645384 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.645389 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.645393 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645397 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.645401 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.645405 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.645412 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.645416 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.645420 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.645425 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.645429 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.645433 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.645437 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.645441 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.645445 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645450 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.645454 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.645458 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.645462 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.645466 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.645479 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645483 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.645487 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.645492 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645496 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.645500 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.645504 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.645509 | controller | period = 900\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.645513 | controller | 1 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0 2026-03-09 19:10:03.645517 | controller | ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier 2026-03-09 19:10:03.645521 | controller | config...'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | Checking watcher-applier 2026-03-09 19:10:03.645525 | controller | config...\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.645530 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.645534 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.645538 | controller | \ logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.645542 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.645546 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.645551 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.645560 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.645564 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.645568 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645573 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.645577 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.645581 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.645585 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.645589 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.645593 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.645598 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.645602 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.645606 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.645610 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.645618 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.645622 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645626 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.645631 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.645635 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.645639 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.645643 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645648 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.645652 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.645663 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645667 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.645671 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.645676 | controller | period = 900\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.645682 | controller | 1 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile 2026-03-09 19:10:03.645686 | controller | = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.645691 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.645695 | controller | \ logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.645699 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.645705 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.645710 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.645714 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.645718 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.645722 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645727 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.645731 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.645735 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.645739 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.645744 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.645750 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.645765 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.645774 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.645780 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.645786 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.645792 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.645798 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645803 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.645809 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.645815 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.645821 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.645825 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645829 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.645833 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.645838 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645842 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.645846 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.645850 | controller | period = 900\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.645855 | controller | 1 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile 2026-03-09 19:10:03.645859 | controller | = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.645874 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.645882 | controller | \ logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.645886 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.645891 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.645895 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.645899 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.645903 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.645908 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645912 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.645916 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.645920 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.645925 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.645929 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.645933 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.645937 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.645945 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.645950 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.645954 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.645958 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.645962 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.645966 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.645971 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.645975 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.645979 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.645983 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.645988 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.645992 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.645996 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646000 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.646005 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.646009 | controller | period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.646013 | controller | 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled 2026-03-09 19:10:03.646021 | controller | = true'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.646025 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.646029 | controller | 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.646033 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.646038 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.646042 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.646053 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.646058 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.646062 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.646067 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.646071 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.646075 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.646079 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.646084 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.646088 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.646092 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.646096 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.646100 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.646105 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.646109 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.646113 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.646117 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.646121 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.646126 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.646130 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.646134 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646138 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.646158 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.646166 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646170 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.646175 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.646181 | controller | period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.646185 | controller | 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n 2026-03-09 19:10:03.646189 | controller | \ logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.646194 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.646198 | controller | 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.646202 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.646206 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.646211 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.646215 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.646219 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.646223 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.646228 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.646232 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.646236 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.646242 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.646254 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.646258 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.646262 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.646267 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.646271 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.646275 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.646280 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.646284 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.646288 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.646294 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.646298 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.646302 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.646306 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646311 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.646315 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.646319 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646326 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.646330 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.646334 | controller | period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.646339 | controller | 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n 2026-03-09 19:10:03.646343 | controller | \ logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.646347 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.646351 | controller | 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.646356 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.646360 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.646364 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.646368 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.646373 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.646377 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.646381 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.646385 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.646390 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.646394 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.646398 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.646402 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.646406 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.646411 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.646415 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.646419 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.646425 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.646454 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.646464 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.646470 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.646475 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.646479 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.646484 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646488 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.646496 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.646500 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646504 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.646508 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.646513 | controller | period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.646517 | controller | 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n 2026-03-09 19:10:03.646521 | controller | \ logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.646525 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.646529 | controller | 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.646534 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.646542 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.646546 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.646550 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.646556 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.646560 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.646565 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.646569 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.646573 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.646577 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.646581 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.646585 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.646590 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.646594 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.646598 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.646602 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.646606 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.646610 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.646615 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.646619 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.646623 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.646631 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.646635 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646640 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.646646 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.646660 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.646667 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.646673 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.646679 | controller | period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.646688 | controller | 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.646695 | controller | ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine 2026-03-09 19:10:03.646700 | controller | config...'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine 2026-03-09 19:10:03.646706 | controller | config...\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.646710 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.646714 | controller | | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.646719 | controller | \ logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | error: Internal error occurred: 2026-03-09 19:10:03.646723 | controller | error executing command in container: container is not created or running\n logger.go:42: 2026-03-09 19:10:03.646727 | controller | 19:03:32 | watcher-tls/2-patch-mtls | ++ echo\n logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.646731 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:03:33 | watcher-tls/2-patch-mtls | running 2026-03-09 19:10:03.646736 | controller | command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n 2026-03-09 19:10:03.646740 | controller | \ # Get pod names for each watcher service\n APIPOD=$(oc get pods -l 2026-03-09 19:10:03.646744 | controller | service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc 2026-03-09 19:10:03.646749 | controller | get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc 2026-03-09 19:10:03.646754 | controller | get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n 2026-03-09 19:10:03.646759 | controller | \ \n # Verify memcached mTLS config parameters in watcher-api config\n 2026-03-09 19:10:03.646765 | controller | \ if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n 2026-03-09 19:10:03.646771 | controller | \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646777 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.646783 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646788 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.646794 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646800 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.646805 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646812 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify 2026-03-09 19:10:03.646816 | controller | mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh 2026-03-09 19:10:03.646822 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.646827 | controller | \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.646831 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.646835 | controller | \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh 2026-03-09 19:10:03.646839 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-09 19:10:03.646844 | controller | \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 2026-03-09 19:10:03.646848 | controller | 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier 2026-03-09 19:10:03.646852 | controller | config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier 2026-03-09 19:10:03.646856 | controller | config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646861 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.646865 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646878 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.646883 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646887 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.646891 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646896 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.646900 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646904 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.646908 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646913 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.646917 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646921 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.646925 | controller | \ exit 1\n fi\n \n # Verify memcached mTLS config parameters 2026-03-09 19:10:03.646929 | controller | in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n 2026-03-09 19:10:03.646934 | controller | \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo 2026-03-09 19:10:03.646938 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646942 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-09 19:10:03.646946 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646951 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-09 19:10:03.646955 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646959 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-09 19:10:03.646965 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646970 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-09 19:10:03.646974 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646979 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.646985 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.646991 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-09 19:10:03.646997 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.647003 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-09 19:10:03.647009 | controller | \ exit 1\n fi\n ]\n logger.go:42: 19:03:33 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.647015 | controller | | + oc project watcher-kuttl-default\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.647021 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.647027 | controller | \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.647033 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.647039 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.647044 | controller | | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n 2026-03-09 19:10:03.647050 | controller | \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n 2026-03-09 19:10:03.647055 | controller | \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.647061 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.647066 | controller | | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:34 2026-03-09 19:10:03.647072 | controller | | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 2026-03-09 19:10:03.647078 | controller | 19:03:34 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n 2026-03-09 19:10:03.647083 | controller | \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n 2026-03-09 19:10:03.647104 | controller | \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.647111 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.647116 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.647122 | controller | \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.647128 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.647134 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.647139 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.647162 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.647175 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.647185 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.647191 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.647197 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.647203 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.647209 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.647215 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.647221 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.647227 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.647233 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.647238 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.647245 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.647250 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.647256 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.647262 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.647269 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.647275 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.647282 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.647288 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.647295 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.647301 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.647308 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.647314 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.647320 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.647326 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.647332 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.647338 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.647344 | controller | period = 900\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.647351 | controller | 1 ']'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile 2026-03-09 19:10:03.647357 | controller | = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.647364 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.647370 | controller | \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.647388 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.647399 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.647405 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.647410 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.647416 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.647422 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.647427 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.647433 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.647439 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.647444 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.647450 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.647456 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.647462 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.647467 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.647473 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.647479 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.647485 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.647490 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.647496 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.647501 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.647508 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.647513 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.647522 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.647528 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.647534 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.647539 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.647545 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.647551 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.647556 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.647562 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.647567 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.647573 | controller | period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.647579 | controller | 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile 2026-03-09 19:10:03.647588 | controller | = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.647594 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.647600 | controller | \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.647606 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.647612 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.647618 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.647624 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.647641 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.647647 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.647653 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.647662 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.647669 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.647675 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.647680 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.647686 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.647692 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.647698 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.647704 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.647710 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.647716 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.647722 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.647728 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.647734 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.647740 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.647746 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.647752 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.647758 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.647764 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.647770 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.647787 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.647793 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.647799 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.647809 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.647816 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.647821 | controller | period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.647825 | controller | 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled 2026-03-09 19:10:03.647830 | controller | = true'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.647834 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.647838 | controller | 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.647843 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.647847 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.647851 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.647855 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.647860 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.647864 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.647868 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.647873 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.647877 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.647891 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.647896 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.647900 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.647904 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.647909 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.647913 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.647920 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.647925 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.647929 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.647933 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.647938 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.647942 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.647946 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.647950 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.647955 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.647962 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.647967 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.647971 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.647975 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.647980 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.647984 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.647988 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.647993 | controller | period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.647997 | controller | 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n 2026-03-09 19:10:03.648001 | controller | \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.648005 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.648010 | controller | 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.648014 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.648019 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.648025 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.648032 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.648038 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.648044 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.648051 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648056 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.648062 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.648068 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.648074 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.648080 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.648085 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.648091 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.648107 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.648112 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.648116 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.648121 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.648125 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648130 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.648136 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.648140 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.648162 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.648167 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.648174 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648179 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.648184 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.648188 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648192 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.648196 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.648201 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.648205 | controller | period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.648209 | controller | 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n 2026-03-09 19:10:03.648214 | controller | \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.648218 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.648227 | controller | 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.648234 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.648240 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.648246 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.648252 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.648258 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.648265 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.648270 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648277 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.648284 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.648288 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.648292 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.648297 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.648301 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.648305 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.648310 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.648316 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.648321 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.648326 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.648332 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648350 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.648358 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.648364 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.648371 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.648377 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.648383 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648389 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.648394 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.648400 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648405 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.648410 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.648414 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.648418 | controller | period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.648423 | controller | 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n 2026-03-09 19:10:03.648427 | controller | \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.648431 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.648435 | controller | 19:03:36 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.648439 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.648444 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.648448 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.648452 | controller | = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.648457 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.648461 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.648465 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648469 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.648473 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.648478 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.648484 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.648489 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.648493 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.648497 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.648501 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.648506 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.648510 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.648514 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.648518 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648522 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.648527 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.648531 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.648537 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.648542 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.648555 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648562 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.648568 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.648574 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648580 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.648586 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.648592 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.648598 | controller | period = 900\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.648604 | controller | 1 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0 2026-03-09 19:10:03.648610 | controller | ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier 2026-03-09 19:10:03.648618 | controller | config...'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | Checking watcher-applier 2026-03-09 19:10:03.648624 | controller | config...\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.648630 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.648636 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.648642 | controller | \ logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.648648 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.648654 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.648660 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.648669 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.648675 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.648681 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648686 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.648692 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.648698 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.648703 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.648707 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.648711 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.648716 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.648720 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.648724 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.648728 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.648733 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.648737 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648742 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.648748 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.648754 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.648760 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.648766 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648772 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.648778 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.648795 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648802 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.648808 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.648814 | controller | period = 900\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.648820 | controller | 1 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile 2026-03-09 19:10:03.648826 | controller | = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.648832 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.648837 | controller | \ logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.648843 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.648853 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.648859 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.648865 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.648871 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.648877 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648883 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.648889 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.648894 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.648900 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.648906 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.648912 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.648917 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.648923 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.648929 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.648939 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.648945 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.648951 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.648957 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.648963 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.648969 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.648975 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.648981 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.648987 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.648993 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.648999 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649005 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.649011 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.649017 | controller | period = 900\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.649023 | controller | 1 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile 2026-03-09 19:10:03.649029 | controller | = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.649046 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.649057 | controller | \ logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.649064 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.649070 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.649076 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.649082 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.649088 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.649094 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649099 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.649105 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.649111 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.649117 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.649123 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.649129 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.649134 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.649140 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.649160 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.649167 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.649172 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.649176 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649180 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.649184 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.649189 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.649193 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.649197 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649201 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.649206 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.649210 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649214 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.649218 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.649223 | controller | period = 900\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.649230 | controller | 1 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled 2026-03-09 19:10:03.649237 | controller | = true'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.649260 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.649264 | controller | 19:03:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.649268 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.649273 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.649277 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.649289 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.649294 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.649298 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649303 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.649307 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.649311 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.649316 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.649320 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.649324 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.649328 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.649336 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.649341 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.649345 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.649349 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.649353 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649358 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.649362 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.649366 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.649370 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.649375 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649379 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.649385 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.649391 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649397 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.649405 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.649414 | controller | period = 900\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.649421 | controller | 1 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n 2026-03-09 19:10:03.649427 | controller | \ logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.649432 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.649436 | controller | 19:03:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.649441 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.649445 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.649449 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.649454 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.649458 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.649462 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649467 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.649473 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.649479 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.649485 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.649501 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.649508 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.649514 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.649519 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.649523 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.649528 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.649532 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.649539 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649543 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.649547 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.649552 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.649556 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.649560 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649564 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.649568 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.649573 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649579 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.649584 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.649588 | controller | period = 900\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.649592 | controller | 1 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n 2026-03-09 19:10:03.649596 | controller | \ logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.649601 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.649605 | controller | 19:03:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.649609 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.649613 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.649618 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.649622 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.649626 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.649631 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649635 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.649639 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.649645 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.649649 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.649653 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.649658 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.649662 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.649666 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.649670 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.649675 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.649679 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.649690 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649695 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.649699 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.649704 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.649708 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.649712 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649718 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.649724 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.649728 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649732 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.649737 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.649742 | controller | period = 900\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.649748 | controller | 1 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n 2026-03-09 19:10:03.649754 | controller | \ logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.649760 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-09 19:10:03.649766 | controller | 19:03:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-09 19:10:03.649772 | controller | transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.649776 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.649788 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.649793 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.649797 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.649801 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649806 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.649810 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.649814 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.649818 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.649823 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.649827 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.649831 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.649835 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.649839 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.649844 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.649848 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.649852 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.649856 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.649861 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.649865 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-09 19:10:03.649873 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.649879 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649885 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.649901 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.649908 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.649914 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.649918 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.649923 | controller | period = 900\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.649927 | controller | 1 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.649931 | controller | ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine 2026-03-09 19:10:03.649935 | controller | config...'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine 2026-03-09 19:10:03.649940 | controller | config...\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-09 19:10:03.649945 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.649952 | controller | | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.649961 | controller | \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.649967 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.649974 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.649980 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.649986 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.649991 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.649997 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650003 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.650012 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.650017 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.650022 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.650026 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.650030 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.650034 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.650039 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.650043 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.650047 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.650054 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.650058 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650064 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.650070 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.650076 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.650082 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.650088 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.650094 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650100 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.650106 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.650112 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650118 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.650135 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.650141 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.650165 | controller | period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.650170 | controller | 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile 2026-03-09 19:10:03.650174 | controller | = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.650178 | controller | | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.650182 | controller | \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.650187 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.650191 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.650195 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.650199 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.650204 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.650208 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650212 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.650218 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.650224 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.650229 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.650235 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.650247 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.650253 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.650259 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.650265 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.650271 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.650277 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.650283 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650288 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.650294 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.650300 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.650306 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.650316 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.650322 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650328 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.650334 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.650340 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650345 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.650351 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.650357 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.650362 | controller | period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.650368 | controller | 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile 2026-03-09 19:10:03.650385 | controller | = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls 2026-03-09 19:10:03.650392 | controller | | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.650398 | controller | \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.650404 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.650410 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.650416 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.650422 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.650428 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.650434 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650440 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.650449 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.650455 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.650461 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.650467 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.650473 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.650479 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.650485 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.650491 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.650497 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.650502 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.650508 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650514 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.650520 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.650526 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.650532 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.650538 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.650544 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650550 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.650556 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.650562 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650568 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.650573 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.650579 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.650585 | controller | period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.650590 | controller | 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled 2026-03-09 19:10:03.650596 | controller | = true'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.650602 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.650608 | controller | \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.650614 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.650620 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.650636 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.650647 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.650653 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.650659 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650665 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.650671 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.650677 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.650683 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.650689 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.650694 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.650700 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.650706 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.650712 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.650722 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.650728 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.650734 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650740 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.650747 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.650752 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.650759 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.650765 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.650771 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650777 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.650783 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.650789 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650799 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.650805 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.650811 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.650817 | controller | period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.650823 | controller | 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n 2026-03-09 19:10:03.650829 | controller | \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.650835 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.650841 | controller | \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.650850 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.650856 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.650862 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.650867 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.650873 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.650891 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650897 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.650901 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.650905 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.650910 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.650914 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.650918 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.650922 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.650927 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.650931 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.650935 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.650939 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.650944 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.650948 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.650952 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.650956 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.650961 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.650965 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.650969 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650973 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.650977 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.650982 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.650986 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.650990 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.650994 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.650999 | controller | period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.651005 | controller | 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n 2026-03-09 19:10:03.651009 | controller | \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.651013 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.651018 | controller | \ logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.651022 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.651026 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.651031 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.651035 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.651039 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.651043 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.651048 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.651052 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.651060 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.651067 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.651073 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.651089 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.651095 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.651100 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.651107 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.651113 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.651118 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.651124 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.651130 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.651136 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.651142 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.651168 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.651172 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.651176 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.651181 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.651185 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.651189 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.651193 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.651201 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.651205 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.651210 | controller | period = 900\n logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.651215 | controller | 1 ']'\n logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n 2026-03-09 19:10:03.651221 | controller | \ logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.651227 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.651233 | controller | \ logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-09 19:10:03.651239 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.651245 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-09 19:10:03.651251 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.651257 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.651263 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.651269 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.651274 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.651280 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.651286 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.651292 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.651297 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.651307 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.651313 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.651319 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.651326 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.651332 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.651349 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.651357 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.651363 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.651369 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.651375 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.651381 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.651387 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-09 19:10:03.651397 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.651403 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-09 19:10:03.651410 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-09 19:10:03.651416 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.651422 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-09 19:10:03.651429 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.651435 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.651445 | controller | period = 900\n logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-09 19:10:03.651451 | controller | 1 ']'\n logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | test step completed 2026-03-09 19:10:03.651457 | controller | 2-patch-mtls\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | 2026-03-09 19:10:03.651463 | controller | starting test step 3-disable-podlevel-tls\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651470 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-09 19:10:03.651476 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/tls/api\", \"value\":{ 2026-03-09 19:10:03.651482 | controller | \"internal\": {}, \"public\": {} }}]'\n ]\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651488 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:03:39 2026-03-09 19:10:03.651493 | controller | | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.651499 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.651505 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.651511 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep 2026-03-09 19:10:03.651517 | controller | watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher 2026-03-09 19:10:03.651522 | controller | -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} 2026-03-09 19:10:03.651528 | controller | watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # check that 2026-03-09 19:10:03.651534 | controller | watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient 2026-03-09 19:10:03.651540 | controller | -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) 2026-03-09 19:10:03.651546 | controller | == 0 ]\n # check that watcher public endpoint does use https\n oc 2026-03-09 19:10:03.651551 | controller | exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim 2026-03-09 19:10:03.651557 | controller | | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container 2026-03-09 19:10:03.651563 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.651568 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.651574 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.651580 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.651586 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.651591 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.651600 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.651606 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.651612 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.651630 | controller | \ ]\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-09 19:10:03.651638 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.651644 | controller | -c Name -c Type\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651650 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651656 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651662 | controller | | ++ grep watcher\n logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651668 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651674 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.651680 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:03:43 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651686 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:43 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651692 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.651698 | controller | \ logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.651704 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651710 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.651716 | controller | \ logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.651722 | controller | ']'\n logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + oc exec 2026-03-09 19:10:03.651732 | controller | -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 2026-03-09 19:10:03.651738 | controller | 19:03:44 | watcher-tls/3-disable-podlevel-tls | + grep internal\n logger.go:42: 2026-03-09 19:10:03.651744 | controller | 19:03:44 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n logger.go:42: 2026-03-09 19:10:03.651750 | controller | 19:03:44 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n logger.go:42: 2026-03-09 19:10:03.651756 | controller | 19:03:46 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 0 ']'\n logger.go:42: 2026-03-09 19:10:03.651762 | controller | 19:03:47 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo 2026-03-09 19:10:03.651768 | controller | pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list 2026-03-09 19:10:03.651791 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec 2026-03-09 19:10:03.651797 | controller | -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type 2026-03-09 19:10:03.651803 | controller | -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice 2026-03-09 19:10:03.651809 | controller | watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc 2026-03-09 19:10:03.651815 | controller | get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n 2026-03-09 19:10:03.651821 | controller | \ # check that watcher internal endpoint does not use https\n oc exec 2026-03-09 19:10:03.651827 | controller | -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | 2026-03-09 19:10:03.651837 | controller | grep internal | [ $(grep -c https) == 0 ]\n # check that watcher public 2026-03-09 19:10:03.651844 | controller | endpoint does use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-09 19:10:03.651850 | controller | endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]\n # 2026-03-09 19:10:03.651856 | controller | If we are running the container locally, skip following test\n if [ \"$(oc 2026-03-09 19:10:03.651862 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.651868 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.651874 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.651880 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.651886 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.651892 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.651898 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.651904 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.651910 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651928 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.651935 | controller | value -c Name -c Type\n logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651942 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651948 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651954 | controller | | ++ grep watcher\n logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651960 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.651966 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651972 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651977 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.651983 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.651989 | controller | \ logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.651995 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652001 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.652005 | controller | \ logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.652009 | controller | ']'\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-09 19:10:03.652018 | controller | \ logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-09 19:10:03.652023 | controller | \ logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.652027 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652031 | controller | | ++ grep -c https\n logger.go:42: 19:03:53 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652038 | controller | | + '[' 1 == 0 ']'\n logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652043 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-09 19:10:03.652047 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-09 19:10:03.652051 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.652055 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-09 19:10:03.652060 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.652064 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-09 19:10:03.652068 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # check that watcher internal endpoint 2026-03-09 19:10:03.652072 | controller | does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-09 19:10:03.652076 | controller | endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n 2026-03-09 19:10:03.652081 | controller | \ # check that watcher public endpoint does use https\n oc exec -n 2026-03-09 19:10:03.652085 | controller | ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep 2026-03-09 19:10:03.652090 | controller | public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, 2026-03-09 19:10:03.652094 | controller | skip following test\n if [ \"$(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.652098 | controller | -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n 2026-03-09 19:10:03.652103 | controller | \ fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.652109 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.652113 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.652118 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.652125 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.652130 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.652134 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.652139 | controller | \ ]\n logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-09 19:10:03.652165 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.652173 | controller | -c Name -c Type\n logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652178 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652182 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652186 | controller | | ++ grep watcher\n logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652190 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652195 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.652199 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652203 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652207 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.652212 | controller | \ logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.652218 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652223 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.652227 | controller | \ logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.652231 | controller | ']'\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + oc exec 2026-03-09 19:10:03.652236 | controller | -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 2026-03-09 19:10:03.652242 | controller | 19:03:58 | watcher-tls/3-disable-podlevel-tls | + grep internal\n logger.go:42: 2026-03-09 19:10:03.652247 | controller | 19:03:58 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n logger.go:42: 2026-03-09 19:10:03.652252 | controller | 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n logger.go:42: 2026-03-09 19:10:03.652258 | controller | 19:04:00 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 0 ']'\n logger.go:42: 2026-03-09 19:10:03.652264 | controller | 19:04:01 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo 2026-03-09 19:10:03.652270 | controller | pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list 2026-03-09 19:10:03.652276 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec 2026-03-09 19:10:03.652282 | controller | -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type 2026-03-09 19:10:03.652288 | controller | -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice 2026-03-09 19:10:03.652294 | controller | watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc 2026-03-09 19:10:03.652300 | controller | get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n 2026-03-09 19:10:03.652304 | controller | \ # check that watcher internal endpoint does not use https\n oc exec 2026-03-09 19:10:03.652309 | controller | -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | 2026-03-09 19:10:03.652313 | controller | grep internal | [ $(grep -c https) == 0 ]\n # check that watcher public 2026-03-09 19:10:03.652317 | controller | endpoint does use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-09 19:10:03.652323 | controller | endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]\n # 2026-03-09 19:10:03.652329 | controller | If we are running the container locally, skip following test\n if [ \"$(oc 2026-03-09 19:10:03.652335 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.652341 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.652347 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.652353 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.652359 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.652365 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.652369 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.652373 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.652377 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652382 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.652399 | controller | value -c Name -c Type\n logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652404 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652408 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652412 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.652416 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652421 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652425 | controller | | ++ grep watcher\n logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652429 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652433 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.652438 | controller | \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.652442 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652446 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.652450 | controller | \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.652455 | controller | ']'\n logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-09 19:10:03.652463 | controller | \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n 2026-03-09 19:10:03.652467 | controller | \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-09 19:10:03.652473 | controller | \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.652477 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652481 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652486 | controller | | + grep public\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652490 | controller | | ++ grep -c https\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652494 | controller | | + grep infra-optim\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652498 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n 2026-03-09 19:10:03.652502 | controller | \ logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n 2026-03-09 19:10:03.652507 | controller | \ logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods 2026-03-09 19:10:03.652511 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-09 19:10:03.652515 | controller | 19:04:10 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n logger.go:42: 2026-03-09 19:10:03.652519 | controller | 19:04:10 | watcher-tls/3-disable-podlevel-tls | + exit 0\n logger.go:42: 19:04:11 2026-03-09 19:10:03.652523 | controller | | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.652528 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.652533 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.652538 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep 2026-03-09 19:10:03.652544 | controller | watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher 2026-03-09 19:10:03.652548 | controller | -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} 2026-03-09 19:10:03.652552 | controller | watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # check that 2026-03-09 19:10:03.652556 | controller | watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient 2026-03-09 19:10:03.652561 | controller | -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) 2026-03-09 19:10:03.652565 | controller | == 0 ]\n # check that watcher public endpoint does use https\n oc 2026-03-09 19:10:03.652569 | controller | exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim 2026-03-09 19:10:03.652574 | controller | | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container 2026-03-09 19:10:03.652578 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.652582 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.652593 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.652598 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.652602 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.652606 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.652610 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.652615 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.652619 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.652623 | controller | \ ]\n logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-09 19:10:03.652627 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.652632 | controller | -c Name -c Type\n logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652636 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652642 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652646 | controller | | ++ grep watcher\n logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652650 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652655 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.652659 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652663 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652667 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.652671 | controller | \ logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.652676 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652680 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.652684 | controller | \ logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.652689 | controller | ']'\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-09 19:10:03.652696 | controller | \ logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-09 19:10:03.652700 | controller | \ logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.652705 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652709 | controller | | ++ grep -c https\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652713 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652717 | controller | | + grep public\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652722 | controller | | ++ grep -c https\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652726 | controller | | + grep infra-optim\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls 2026-03-09 19:10:03.652730 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n 2026-03-09 19:10:03.652734 | controller | \ logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n 2026-03-09 19:10:03.652739 | controller | \ logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods 2026-03-09 19:10:03.652744 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-09 19:10:03.652750 | controller | 19:04:20 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n logger.go:42: 2026-03-09 19:10:03.652756 | controller | 19:04:20 | watcher-tls/3-disable-podlevel-tls | + exit 0\n logger.go:42: 19:04:20 2026-03-09 19:10:03.652762 | controller | | watcher-tls/3-disable-podlevel-tls | test step completed 3-disable-podlevel-tls\n 2026-03-09 19:10:03.652768 | controller | \ logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | starting test 2026-03-09 19:10:03.652774 | controller | step 4-deploy-without-route\n logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652779 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-09 19:10:03.652783 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{\"service\": 2026-03-09 19:10:03.652796 | controller | { \"internal\": {}, \"public\": { \"metadata\": { \"annotations\": { \"metallb.universe.tf/address-pool\": 2026-03-09 19:10:03.652800 | controller | \"ctlplane\", \"metallb.universe.tf/allow-shared-ip\": \"ctlplane\" } }, \"spec\": 2026-03-09 19:10:03.652805 | controller | { \"type\": \"LoadBalancer\" } } } }}]'\n ]\n logger.go:42: 19:04:20 | 2026-03-09 19:10:03.652809 | controller | watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.652813 | controller | patched\n logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | running 2026-03-09 19:10:03.652817 | controller | command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-09 19:10:03.652822 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-09 19:10:03.652826 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.652830 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-09 19:10:03.652835 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.652839 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-09 19:10:03.652843 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-09 19:10:03.652850 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.652855 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.652861 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.652865 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.652870 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.652875 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.652880 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.652884 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.652888 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.652892 | controller | \ ]\n logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | ++ 2026-03-09 19:10:03.652897 | controller | grep -c '^watcher'\n logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652901 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.652905 | controller | value -c Name -c Type\n logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652909 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652914 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.652918 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652922 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652926 | controller | | ++ grep watcher\n logger.go:42: 19:04:24 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652931 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:24 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652935 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.652939 | controller | \ logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.652943 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.652948 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.652952 | controller | \ logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.652956 | controller | ']'\n logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | ++ oc get 2026-03-09 19:10:03.652960 | controller | pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-09 19:10:03.652965 | controller | 19:04:25 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n logger.go:42: 2026-03-09 19:10:03.652969 | controller | 19:04:25 | watcher-tls/4-deploy-without-route | + exit 0\n logger.go:42: 19:04:26 2026-03-09 19:10:03.652973 | controller | | watcher-tls/4-deploy-without-route | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.652977 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.652983 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.652998 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep 2026-03-09 19:10:03.653005 | controller | watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher 2026-03-09 19:10:03.653011 | controller | -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} 2026-03-09 19:10:03.653017 | controller | watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are 2026-03-09 19:10:03.653027 | controller | running the container locally, skip following test\n if [ \"$(oc get pods 2026-03-09 19:10:03.653031 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" 2026-03-09 19:10:03.653036 | controller | ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc 2026-03-09 19:10:03.653040 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.653044 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.653049 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.653053 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.653057 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.653061 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.653066 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653070 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.653076 | controller | value -c Name -c Type\n logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653081 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653085 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653089 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.653094 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653098 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653102 | controller | | ++ grep watcher\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653106 | controller | | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653111 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.653115 | controller | \ logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + '[' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.653119 | controller | == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route 2026-03-09 19:10:03.653123 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.653128 | controller | \ logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.653132 | controller | ']'\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get 2026-03-09 19:10:03.653136 | controller | pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-09 19:10:03.653140 | controller | 19:04:30 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n logger.go:42: 2026-03-09 19:10:03.653158 | controller | 19:04:30 | watcher-tls/4-deploy-without-route | + exit 0\n logger.go:42: 19:04:30 2026-03-09 19:10:03.653165 | controller | | watcher-tls/4-deploy-without-route | test step completed 4-deploy-without-route\n 2026-03-09 19:10:03.653170 | controller | \ logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | starting test step 5-disable-tls\n 2026-03-09 19:10:03.653174 | controller | \ logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | running command: [sh -c 2026-03-09 19:10:03.653178 | controller | oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\", 2026-03-09 19:10:03.653185 | controller | \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{}}]'\n ]\n logger.go:42: 2026-03-09 19:10:03.653189 | controller | 19:04:30 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.653194 | controller | patched\n logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | running command: 2026-03-09 19:10:03.653198 | controller | [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-09 19:10:03.653202 | controller | service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc 2026-03-09 19:10:03.653206 | controller | exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.653219 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} 2026-03-09 19:10:03.653223 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-09 19:10:03.653227 | controller | -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-09 19:10:03.653232 | controller | ]\n # If we are running the container locally, skip following test\n if 2026-03-09 19:10:03.653236 | controller | [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.653240 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.653244 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.653249 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.653253 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.653257 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.653261 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.653266 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.653270 | controller | variables found.\"\n fi\n # check that no watcher endpoint uses https\n 2026-03-09 19:10:03.653274 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.653278 | controller | infra-optim | [ $(grep -c https) == 0 ]\n ]\n logger.go:42: 19:04:31 | 2026-03-09 19:10:03.653283 | controller | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default openstackclient -- 2026-03-09 19:10:03.653287 | controller | openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:31 | watcher-tls/5-disable-tls 2026-03-09 19:10:03.653291 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:04:33 | watcher-tls/5-disable-tls 2026-03-09 19:10:03.653295 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ 2026-03-09 19:10:03.653302 | controller | grep watcher\n logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ awk '{print 2026-03-09 19:10:03.653306 | controller | $1}'\n logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.653310 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 2026-03-09 19:10:03.653314 | controller | 19:04:35 | watcher-tls/5-disable-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n 2026-03-09 19:10:03.653319 | controller | \ logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.653323 | controller | keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n logger.go:42: 19:04:35 2026-03-09 19:10:03.653329 | controller | | watcher-tls/5-disable-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.653333 | controller | ']'\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.653339 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:04:35 2026-03-09 19:10:03.653345 | controller | | watcher-tls/5-disable-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.653349 | controller | ']'\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get pods -n 2026-03-09 19:10:03.653354 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-09 19:10:03.653358 | controller | 19:04:35 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n logger.go:42: 19:04:35 2026-03-09 19:10:03.653362 | controller | | watcher-tls/5-disable-tls | + exit 0\n logger.go:42: 19:04:36 | watcher-tls/5-disable-tls 2026-03-09 19:10:03.653366 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-09 19:10:03.653370 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-09 19:10:03.653375 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.653379 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-09 19:10:03.653383 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.653387 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-09 19:10:03.653393 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-09 19:10:03.653399 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.653415 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.653422 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.653440 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.653447 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.653453 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.653459 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.653464 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.653470 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.653475 | controller | \ # check that no watcher endpoint uses https\n oc exec -n ${NAMESPACE} 2026-03-09 19:10:03.653480 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-09 19:10:03.653486 | controller | == 0 ]\n ]\n logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | + oc 2026-03-09 19:10:03.653492 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.653497 | controller | -c Name -c Type\n logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | ++ grep 2026-03-09 19:10:03.653502 | controller | -c '^watcher'\n logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | + '[' 1 2026-03-09 19:10:03.653506 | controller | == 1 ']'\n logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ grep watcher\n 2026-03-09 19:10:03.653510 | controller | \ logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ awk '{print $1}'\n logger.go:42: 2026-03-09 19:10:03.653515 | controller | 19:04:38 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.653519 | controller | -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:04:40 2026-03-09 19:10:03.653523 | controller | | watcher-tls/5-disable-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 2026-03-09 19:10:03.653530 | controller | 19:04:40 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default keystoneservice 2026-03-09 19:10:03.653535 | controller | watcher -o 'jsonpath={.status.serviceID}'\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls 2026-03-09 19:10:03.653539 | controller | | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n 2026-03-09 19:10:03.653543 | controller | \ logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.653547 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:04:40 2026-03-09 19:10:03.653552 | controller | | watcher-tls/5-disable-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.653556 | controller | ']'\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get pods -n 2026-03-09 19:10:03.653563 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-09 19:10:03.653568 | controller | 19:04:40 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n logger.go:42: 19:04:40 2026-03-09 19:10:03.653572 | controller | | watcher-tls/5-disable-tls | + exit 0\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls 2026-03-09 19:10:03.653576 | controller | | test step completed 5-disable-tls\n logger.go:42: 19:04:40 | watcher-tls/6-cleanup-watcher 2026-03-09 19:10:03.653580 | controller | | starting test step 6-cleanup-watcher\n logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher 2026-03-09 19:10:03.653585 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.653589 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.653593 | controller | ]\n ]\n logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | + oc 2026-03-09 19:10:03.653597 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.653601 | controller | -c Name -c Type\n logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | ++ 2026-03-09 19:10:03.653606 | controller | grep -c '^watcher'\n logger.go:42: 19:04:43 | watcher-tls/6-cleanup-watcher | 2026-03-09 19:10:03.653610 | controller | + '[' 1 == 0 ']'\n logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | running 2026-03-09 19:10:03.653614 | controller | command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.653618 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.653622 | controller | ]\n ]\n logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | + oc 2026-03-09 19:10:03.653627 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.653631 | controller | -c Name -c Type\n logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | ++ 2026-03-09 19:10:03.653635 | controller | grep -c '^watcher'\n logger.go:42: 19:04:47 | watcher-tls/6-cleanup-watcher | 2026-03-09 19:10:03.653639 | controller | + '[' 0 == 0 ']'\n logger.go:42: 19:04:47 | watcher-tls/6-cleanup-watcher | test 2026-03-09 19:10:03.653643 | controller | step completed 6-cleanup-watcher\n logger.go:42: 19:04:47 | watcher-tls/7-cleanup-certs 2026-03-09 19:10:03.653656 | controller | | starting test step 7-cleanup-certs\n logger.go:42: 19:04:47 | watcher-tls/7-cleanup-certs 2026-03-09 19:10:03.653660 | controller | | test step completed 7-cleanup-certs\n logger.go:42: 19:04:47 | watcher-tls 2026-03-09 19:10:03.653664 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-rmquser\n logger.go:42: 2026-03-09 19:10:03.653669 | controller | 19:04:47 | watcher-rmquser | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-09 19:10:03.653673 | controller | \ logger.go:42: 19:04:47 | watcher-rmquser/0-cleanup-watcher | starting test step 2026-03-09 19:10:03.653677 | controller | 0-cleanup-watcher\n logger.go:42: 19:04:47 | watcher-rmquser/0-cleanup-watcher 2026-03-09 19:10:03.653683 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 19:04:47 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.653688 | controller | | starting test step 1-deploy\n logger.go:42: 19:04:47 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.653692 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:04:47 2026-03-09 19:10:03.653696 | controller | | watcher-rmquser/1-deploy | running command: [sh -c set -euxo pipefail\n \n 2026-03-09 19:10:03.653700 | controller | \ # Wait for Watcher to be Ready\n kubectl wait --for=condition=Ready 2026-03-09 19:10:03.653704 | controller | watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n \n # Verify WatcherNotificationTransportURLReady 2026-03-09 19:10:03.653709 | controller | condition exists and is True\n kubectl get watcher watcher-kuttl -n $NAMESPACE 2026-03-09 19:10:03.653713 | controller | -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}' 2026-03-09 19:10:03.653717 | controller | | grep -q \"True\"\n echo \"WatcherNotificationTransportURLReady condition 2026-03-09 19:10:03.653721 | controller | is True\"\n \n # Count TransportURL CRs - should be exactly 2 (one 2026-03-09 19:10:03.653725 | controller | for messaging, one for notifications)\n transport_count=$(kubectl get transporturl 2026-03-09 19:10:03.653730 | controller | -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n notification_transport_count=$(kubectl 2026-03-09 19:10:03.653734 | controller | get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\" 2026-03-09 19:10:03.653738 | controller | | wc -l)\n \n if [ \"$transport_count\" -ne \"1\" ]; then\n echo 2026-03-09 19:10:03.653742 | controller | \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n exit 2026-03-09 19:10:03.653746 | controller | 1\n fi\n \n if [ \"$notification_transport_count\" -ne \"1\" 2026-03-09 19:10:03.653751 | controller | ]; then\n echo \"Expected 1 notification-transport TransportURL, found 2026-03-09 19:10:03.653755 | controller | $notification_transport_count\"\n exit 1\n fi\n \n echo 2026-03-09 19:10:03.653761 | controller | \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n 2026-03-09 19:10:03.653765 | controller | \ \n # Verify watcher-transport has correct user and vhost\n transport_user=$(kubectl 2026-03-09 19:10:03.653770 | controller | get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n 2026-03-09 19:10:03.653774 | controller | \ transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-09 19:10:03.653778 | controller | -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n if [ \"$transport_user\" != 2026-03-09 19:10:03.653782 | controller | \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport username 'watcher-rpc', 2026-03-09 19:10:03.653786 | controller | found '$transport_user'\"\n exit 1\n fi\n if [ \"$transport_vhost\" 2026-03-09 19:10:03.653791 | controller | != \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport vhost 'watcher-rpc', 2026-03-09 19:10:03.653795 | controller | found '$transport_vhost'\"\n exit 1\n fi\n echo \"Watcher 2026-03-09 19:10:03.653799 | controller | transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n \n # 2026-03-09 19:10:03.653803 | controller | Verify notification-transport has correct user and vhost\n notif_user=$(kubectl 2026-03-09 19:10:03.653807 | controller | get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE 2026-03-09 19:10:03.653812 | controller | -o jsonpath='{.spec.username}')\n notif_vhost=$(kubectl get transporturl 2026-03-09 19:10:03.653816 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n 2026-03-09 19:10:03.653820 | controller | \ if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n echo 2026-03-09 19:10:03.653828 | controller | \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n 2026-03-09 19:10:03.653833 | controller | \ exit 1\n fi\n if [ \"$notif_vhost\" != \"watcher-notifications\" 2026-03-09 19:10:03.653837 | controller | ]; then\n echo \"Expected notification-transport vhost 'watcher-notifications', 2026-03-09 19:10:03.653841 | controller | found '$notif_vhost'\"\n exit 1\n fi\n echo \"Notification 2026-03-09 19:10:03.653852 | controller | transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n 2026-03-09 19:10:03.653856 | controller | \ \n # Verify that watcher.conf contains the notifications transport_url\n 2026-03-09 19:10:03.653861 | controller | \ WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\" 2026-03-09 19:10:03.653865 | controller | -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n if 2026-03-09 19:10:03.653869 | controller | [ -z \"${WATCHER_API_POD}\" ]; then\n echo \"No watcher-api pod found\"\n 2026-03-09 19:10:03.653873 | controller | \ exit 1\n fi\n # Verify RPC transport_url in DEFAULT section\n 2026-03-09 19:10:03.653878 | controller | \ rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api 2026-03-09 19:10:03.653882 | controller | -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A 2026-03-09 19:10:03.653886 | controller | 50 | grep 'transport_url' | head -1 || true)\n if [ -z \"$rpc_transport_url\" 2026-03-09 19:10:03.653890 | controller | ]; then\n echo \"transport_url not found in DEFAULT section\"\n exit 2026-03-09 19:10:03.653894 | controller | 1\n fi\n echo \"Found RPC transport_url: $rpc_transport_url\"\n \n 2026-03-09 19:10:03.653899 | controller | \ # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n 2026-03-09 19:10:03.653903 | controller | \ if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n echo 2026-03-09 19:10:03.653907 | controller | \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n exit 2026-03-09 19:10:03.653911 | controller | 1\n fi\n echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n 2026-03-09 19:10:03.653915 | controller | \ \n # Verify the RPC transport_url contains the correct username (watcher-rpc)\n 2026-03-09 19:10:03.653920 | controller | \ if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n echo 2026-03-09 19:10:03.653924 | controller | \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n exit 2026-03-09 19:10:03.653928 | controller | 1\n fi\n echo \"Successfully verified username 'watcher-rpc' in RPC 2026-03-09 19:10:03.653932 | controller | transport_url\"\n \n # Verify oslo_messaging_notifications section 2026-03-09 19:10:03.653936 | controller | has transport_url configured\n notif_transport_url=$(kubectl exec -n $NAMESPACE 2026-03-09 19:10:03.653943 | controller | ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf 2026-03-09 19:10:03.653947 | controller | | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n 2026-03-09 19:10:03.653951 | controller | \ if [ -z \"$notif_transport_url\" ]; then\n echo \"transport_url 2026-03-09 19:10:03.653955 | controller | not found in oslo_messaging_notifications section\"\n exit 1\n fi\n 2026-03-09 19:10:03.653959 | controller | \ echo \"Found notifications transport_url: $notif_transport_url\"\n \n 2026-03-09 19:10:03.653964 | controller | \ # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n 2026-03-09 19:10:03.653968 | controller | \ if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n 2026-03-09 19:10:03.653972 | controller | \ echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n 2026-03-09 19:10:03.653976 | controller | \ exit 1\n fi\n echo \"Successfully verified vhost 'watcher-notifications' 2026-03-09 19:10:03.653982 | controller | in notifications transport_url\"\n \n # Verify the notifications transport_url 2026-03-09 19:10:03.653987 | controller | contains the correct username (watcher-notifications)\n if ! echo \"$notif_transport_url\" 2026-03-09 19:10:03.653991 | controller | | grep -q 'watcher-notifications:'; then\n echo \"Notifications transport_url 2026-03-09 19:10:03.653995 | controller | does not contain expected username 'watcher-notifications:'\"\n exit 1\n 2026-03-09 19:10:03.653999 | controller | \ fi\n echo \"Successfully verified username 'watcher-notifications' 2026-03-09 19:10:03.654004 | controller | in notifications transport_url\"\n \n exit 0\n ]\n logger.go:42: 2026-03-09 19:10:03.654009 | controller | 19:04:47 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl 2026-03-09 19:10:03.654015 | controller | -n watcher-kuttl-default --timeout=300s\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654021 | controller | | watcher.watcher.openstack.org/watcher-kuttl condition met\n logger.go:42: 19:05:12 2026-03-09 19:10:03.654028 | controller | | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default 2026-03-09 19:10:03.654034 | controller | -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n 2026-03-09 19:10:03.654039 | controller | \ logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + grep -q True\n logger.go:42: 2026-03-09 19:10:03.654045 | controller | 19:05:12 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady 2026-03-09 19:10:03.654060 | controller | condition is True'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady 2026-03-09 19:10:03.654066 | controller | condition is True\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl 2026-03-09 19:10:03.654071 | controller | get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 19:05:12 | 2026-03-09 19:10:03.654077 | controller | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654083 | controller | | ++ grep watcher-kuttl-watcher-transport\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654088 | controller | | + transport_count=1\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ 2026-03-09 19:10:03.654093 | controller | kubectl get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 19:05:12 2026-03-09 19:10:03.654099 | controller | | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654104 | controller | | ++ grep watcher-kuttl-watcher-notification\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654109 | controller | | + notification_transport_count=1\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654113 | controller | | + '[' 1 -ne 1 ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + '[' 2026-03-09 19:10:03.654118 | controller | 1 -ne 1 ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo 'Correctly 2026-03-09 19:10:03.654122 | controller | found 2 TransportURLs (separate clusters: transport and notification)'\n logger.go:42: 2026-03-09 19:10:03.654126 | controller | 19:05:12 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate 2026-03-09 19:10:03.654130 | controller | clusters: transport and notification)\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654134 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default 2026-03-09 19:10:03.654138 | controller | -o 'jsonpath={.spec.username}'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654143 | controller | | + transport_user=watcher-rpc\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654164 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default 2026-03-09 19:10:03.654169 | controller | -o 'jsonpath={.spec.vhost}'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654177 | controller | | + transport_vhost=watcher-rpc\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654181 | controller | | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654186 | controller | | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654190 | controller | | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n 2026-03-09 19:10:03.654194 | controller | \ logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Watcher transport has correct 2026-03-09 19:10:03.654198 | controller | user (watcher-rpc) and vhost (watcher-rpc)\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654203 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-09 19:10:03.654207 | controller | -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n logger.go:42: 19:05:12 2026-03-09 19:10:03.654211 | controller | | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n logger.go:42: 2026-03-09 19:10:03.654215 | controller | 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-09 19:10:03.654219 | controller | -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n logger.go:42: 19:05:12 2026-03-09 19:10:03.654226 | controller | | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n logger.go:42: 2026-03-09 19:10:03.654231 | controller | 19:05:12 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications 2026-03-09 19:10:03.654235 | controller | ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + '[' watcher-notifications 2026-03-09 19:10:03.654239 | controller | '!=' watcher-notifications ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654244 | controller | | + echo 'Notification transport has correct user (watcher-notifications) and vhost 2026-03-09 19:10:03.654248 | controller | (watcher-notifications)'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654252 | controller | | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n 2026-03-09 19:10:03.654256 | controller | \ logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default 2026-03-09 19:10:03.654260 | controller | -l service=watcher-api -o custom-columns=:metadata.name --no-headers\n logger.go:42: 2026-03-09 19:10:03.654265 | controller | 19:05:12 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654269 | controller | | ++ grep -v '^$'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0\n 2026-03-09 19:10:03.654273 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + '[' -z watcher-kuttl-api-0 2026-03-09 19:10:03.654285 | controller | ']'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]' 2026-03-09 19:10:03.654289 | controller | -A 50\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep transport_url\n 2026-03-09 19:10:03.654293 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-09 19:10:03.654298 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.654302 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 2026-03-09 19:10:03.654306 | controller | 19:05:13 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-09 19:10:03.654311 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + '[' -z 'transport_url 2026-03-09 19:10:03.654315 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1' 2026-03-09 19:10:03.654321 | controller | ']'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url: 2026-03-09 19:10:03.654325 | controller | transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-09 19:10:03.654330 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Found RPC transport_url: 2026-03-09 19:10:03.654334 | controller | transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1\n 2026-03-09 19:10:03.654338 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-09 19:10:03.654342 | controller | rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-09 19:10:03.654347 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n 2026-03-09 19:10:03.654351 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-09 19:10:03.654355 | controller | vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 19:05:13 | 2026-03-09 19:10:03.654361 | controller | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n 2026-03-09 19:10:03.654365 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n 2026-03-09 19:10:03.654369 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-09 19:10:03.654374 | controller | rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-09 19:10:03.654378 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-09 19:10:03.654382 | controller | username '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 19:05:13 2026-03-09 19:10:03.654386 | controller | | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC 2026-03-09 19:10:03.654391 | controller | transport_url\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ kubectl 2026-03-09 19:10:03.654395 | controller | exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.654399 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep -A 5 '\\[oslo_messaging_notifications\\]'\n 2026-03-09 19:10:03.654403 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep transport_url\n 2026-03-09 19:10:03.654407 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + notif_transport_url='transport_url 2026-03-09 19:10:03.654412 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-09 19:10:03.654416 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + '[' -z 'transport_url 2026-03-09 19:10:03.654420 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1' 2026-03-09 19:10:03.654426 | controller | ']'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Found notifications 2026-03-09 19:10:03.654431 | controller | transport_url: transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-09 19:10:03.654435 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Found notifications transport_url: 2026-03-09 19:10:03.654439 | controller | transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1\n 2026-03-09 19:10:03.654444 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-09 19:10:03.654455 | controller | rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-09 19:10:03.654461 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n 2026-03-09 19:10:03.654465 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-09 19:10:03.654470 | controller | vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 2026-03-09 19:10:03.654474 | controller | 19:05:13 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications' 2026-03-09 19:10:03.654478 | controller | in notifications transport_url\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654482 | controller | | + echo 'transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-09 19:10:03.654487 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:\n 2026-03-09 19:10:03.654491 | controller | \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-09 19:10:03.654495 | controller | username '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 2026-03-09 19:10:03.654499 | controller | 19:05:13 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications' 2026-03-09 19:10:03.654505 | controller | in notifications transport_url\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654509 | controller | | + exit 0\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | running command: 2026-03-09 19:10:03.654514 | controller | [sh -c set -euxo pipefail\n \n # Wait for Watcher to be Ready\n kubectl 2026-03-09 19:10:03.654518 | controller | wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n 2026-03-09 19:10:03.654522 | controller | \ \n # Verify WatcherNotificationTransportURLReady condition exists 2026-03-09 19:10:03.654526 | controller | and is True\n kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}' 2026-03-09 19:10:03.654531 | controller | | grep -q \"True\"\n echo \"WatcherNotificationTransportURLReady condition 2026-03-09 19:10:03.654535 | controller | is True\"\n \n # Count TransportURL CRs - should be exactly 2 (one 2026-03-09 19:10:03.654539 | controller | for messaging, one for notifications)\n transport_count=$(kubectl get transporturl 2026-03-09 19:10:03.654543 | controller | -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n notification_transport_count=$(kubectl 2026-03-09 19:10:03.654547 | controller | get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\" 2026-03-09 19:10:03.654552 | controller | | wc -l)\n \n if [ \"$transport_count\" -ne \"1\" ]; then\n echo 2026-03-09 19:10:03.654556 | controller | \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n exit 2026-03-09 19:10:03.654560 | controller | 1\n fi\n \n if [ \"$notification_transport_count\" -ne \"1\" 2026-03-09 19:10:03.654564 | controller | ]; then\n echo \"Expected 1 notification-transport TransportURL, found 2026-03-09 19:10:03.654568 | controller | $notification_transport_count\"\n exit 1\n fi\n \n echo 2026-03-09 19:10:03.654573 | controller | \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n 2026-03-09 19:10:03.654577 | controller | \ \n # Verify watcher-transport has correct user and vhost\n transport_user=$(kubectl 2026-03-09 19:10:03.654581 | controller | get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n 2026-03-09 19:10:03.654585 | controller | \ transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-09 19:10:03.654590 | controller | -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n if [ \"$transport_user\" != 2026-03-09 19:10:03.654594 | controller | \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport username 'watcher-rpc', 2026-03-09 19:10:03.654600 | controller | found '$transport_user'\"\n exit 1\n fi\n if [ \"$transport_vhost\" 2026-03-09 19:10:03.654604 | controller | != \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport vhost 'watcher-rpc', 2026-03-09 19:10:03.654608 | controller | found '$transport_vhost'\"\n exit 1\n fi\n echo \"Watcher 2026-03-09 19:10:03.654613 | controller | transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n \n # 2026-03-09 19:10:03.654617 | controller | Verify notification-transport has correct user and vhost\n notif_user=$(kubectl 2026-03-09 19:10:03.654621 | controller | get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE 2026-03-09 19:10:03.654625 | controller | -o jsonpath='{.spec.username}')\n notif_vhost=$(kubectl get transporturl 2026-03-09 19:10:03.654630 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n 2026-03-09 19:10:03.654634 | controller | \ if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n echo 2026-03-09 19:10:03.654644 | controller | \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n 2026-03-09 19:10:03.654649 | controller | \ exit 1\n fi\n if [ \"$notif_vhost\" != \"watcher-notifications\" 2026-03-09 19:10:03.654653 | controller | ]; then\n echo \"Expected notification-transport vhost 'watcher-notifications', 2026-03-09 19:10:03.654658 | controller | found '$notif_vhost'\"\n exit 1\n fi\n echo \"Notification 2026-03-09 19:10:03.654662 | controller | transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n 2026-03-09 19:10:03.654666 | controller | \ \n # Verify that watcher.conf contains the notifications transport_url\n 2026-03-09 19:10:03.654670 | controller | \ WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\" 2026-03-09 19:10:03.654674 | controller | -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n if 2026-03-09 19:10:03.654679 | controller | [ -z \"${WATCHER_API_POD}\" ]; then\n echo \"No watcher-api pod found\"\n 2026-03-09 19:10:03.654683 | controller | \ exit 1\n fi\n # Verify RPC transport_url in DEFAULT section\n 2026-03-09 19:10:03.654687 | controller | \ rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api 2026-03-09 19:10:03.654693 | controller | -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A 2026-03-09 19:10:03.654697 | controller | 50 | grep 'transport_url' | head -1 || true)\n if [ -z \"$rpc_transport_url\" 2026-03-09 19:10:03.654701 | controller | ]; then\n echo \"transport_url not found in DEFAULT section\"\n exit 2026-03-09 19:10:03.654706 | controller | 1\n fi\n echo \"Found RPC transport_url: $rpc_transport_url\"\n \n 2026-03-09 19:10:03.654710 | controller | \ # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n 2026-03-09 19:10:03.654714 | controller | \ if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n echo 2026-03-09 19:10:03.654718 | controller | \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n exit 2026-03-09 19:10:03.654722 | controller | 1\n fi\n echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n 2026-03-09 19:10:03.654727 | controller | \ \n # Verify the RPC transport_url contains the correct username (watcher-rpc)\n 2026-03-09 19:10:03.654731 | controller | \ if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n echo 2026-03-09 19:10:03.654735 | controller | \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n exit 2026-03-09 19:10:03.654739 | controller | 1\n fi\n echo \"Successfully verified username 'watcher-rpc' in RPC 2026-03-09 19:10:03.654744 | controller | transport_url\"\n \n # Verify oslo_messaging_notifications section 2026-03-09 19:10:03.654750 | controller | has transport_url configured\n notif_transport_url=$(kubectl exec -n $NAMESPACE 2026-03-09 19:10:03.654754 | controller | ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf 2026-03-09 19:10:03.654758 | controller | | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n 2026-03-09 19:10:03.654762 | controller | \ if [ -z \"$notif_transport_url\" ]; then\n echo \"transport_url 2026-03-09 19:10:03.654767 | controller | not found in oslo_messaging_notifications section\"\n exit 1\n fi\n 2026-03-09 19:10:03.654771 | controller | \ echo \"Found notifications transport_url: $notif_transport_url\"\n \n 2026-03-09 19:10:03.654775 | controller | \ # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n 2026-03-09 19:10:03.654779 | controller | \ if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n 2026-03-09 19:10:03.654784 | controller | \ echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n 2026-03-09 19:10:03.654788 | controller | \ exit 1\n fi\n echo \"Successfully verified vhost 'watcher-notifications' 2026-03-09 19:10:03.654792 | controller | in notifications transport_url\"\n \n # Verify the notifications transport_url 2026-03-09 19:10:03.654796 | controller | contains the correct username (watcher-notifications)\n if ! echo \"$notif_transport_url\" 2026-03-09 19:10:03.654800 | controller | | grep -q 'watcher-notifications:'; then\n echo \"Notifications transport_url 2026-03-09 19:10:03.654805 | controller | does not contain expected username 'watcher-notifications:'\"\n exit 1\n 2026-03-09 19:10:03.654809 | controller | \ fi\n echo \"Successfully verified username 'watcher-notifications' 2026-03-09 19:10:03.654813 | controller | in notifications transport_url\"\n \n exit 0\n ]\n logger.go:42: 2026-03-09 19:10:03.654817 | controller | 19:05:14 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl 2026-03-09 19:10:03.654822 | controller | -n watcher-kuttl-default --timeout=300s\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654826 | controller | | watcher.watcher.openstack.org/watcher-kuttl condition met\n logger.go:42: 19:05:14 2026-03-09 19:10:03.654837 | controller | | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default 2026-03-09 19:10:03.654842 | controller | -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n 2026-03-09 19:10:03.654846 | controller | \ logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + grep -q True\n logger.go:42: 2026-03-09 19:10:03.654850 | controller | 19:05:14 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady 2026-03-09 19:10:03.654855 | controller | condition is True'\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady 2026-03-09 19:10:03.654859 | controller | condition is True\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ grep 2026-03-09 19:10:03.654863 | controller | watcher-kuttl-watcher-transport\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654867 | controller | | ++ kubectl get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 2026-03-09 19:10:03.654872 | controller | 19:05:14 | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654876 | controller | | + transport_count=1\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ 2026-03-09 19:10:03.654880 | controller | wc -l\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification\n 2026-03-09 19:10:03.654885 | controller | \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.654889 | controller | -n watcher-kuttl-default -o name\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654895 | controller | | + notification_transport_count=1\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654901 | controller | | + '[' 1 -ne 1 ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' 2026-03-09 19:10:03.654906 | controller | 1 -ne 1 ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo 'Correctly 2026-03-09 19:10:03.654911 | controller | found 2 TransportURLs (separate clusters: transport and notification)'\n logger.go:42: 2026-03-09 19:10:03.654915 | controller | 19:05:15 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate 2026-03-09 19:10:03.654919 | controller | clusters: transport and notification)\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654923 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default 2026-03-09 19:10:03.654928 | controller | -o 'jsonpath={.spec.username}'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654932 | controller | | + transport_user=watcher-rpc\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654936 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default 2026-03-09 19:10:03.654940 | controller | -o 'jsonpath={.spec.vhost}'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654944 | controller | | + transport_vhost=watcher-rpc\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654949 | controller | | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654953 | controller | | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654957 | controller | | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n 2026-03-09 19:10:03.654961 | controller | \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Watcher transport has correct 2026-03-09 19:10:03.654965 | controller | user (watcher-rpc) and vhost (watcher-rpc)\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.654970 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-09 19:10:03.654974 | controller | -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n logger.go:42: 19:05:15 2026-03-09 19:10:03.654978 | controller | | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n logger.go:42: 2026-03-09 19:10:03.654982 | controller | 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-09 19:10:03.654986 | controller | -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n logger.go:42: 19:05:15 2026-03-09 19:10:03.654992 | controller | | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n logger.go:42: 2026-03-09 19:10:03.654996 | controller | 19:05:15 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications 2026-03-09 19:10:03.655002 | controller | ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' watcher-notifications 2026-03-09 19:10:03.655007 | controller | '!=' watcher-notifications ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.655013 | controller | | + echo 'Notification transport has correct user (watcher-notifications) and vhost 2026-03-09 19:10:03.655019 | controller | (watcher-notifications)'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.655025 | controller | | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n 2026-03-09 19:10:03.655031 | controller | \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep -v '^$'\n logger.go:42: 2026-03-09 19:10:03.655046 | controller | 19:05:15 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.655052 | controller | | ++ kubectl get pods -n watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name 2026-03-09 19:10:03.655058 | controller | --no-headers\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0\n 2026-03-09 19:10:03.655068 | controller | \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' -z watcher-kuttl-api-0 2026-03-09 19:10:03.655072 | controller | ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]' 2026-03-09 19:10:03.655077 | controller | -A 50\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep transport_url\n 2026-03-09 19:10:03.655081 | controller | \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-09 19:10:03.655085 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.655089 | controller | \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 2026-03-09 19:10:03.655094 | controller | 19:05:16 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-09 19:10:03.655098 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + '[' -z 'transport_url 2026-03-09 19:10:03.655102 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1' 2026-03-09 19:10:03.655106 | controller | ']'\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url: 2026-03-09 19:10:03.655111 | controller | transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-09 19:10:03.655115 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Found RPC transport_url: 2026-03-09 19:10:03.655119 | controller | transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1\n 2026-03-09 19:10:03.655124 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-09 19:10:03.655130 | controller | rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-09 19:10:03.655135 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n 2026-03-09 19:10:03.655139 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-09 19:10:03.655157 | controller | vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 19:05:16 | 2026-03-09 19:10:03.655164 | controller | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n 2026-03-09 19:10:03.655168 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-09 19:10:03.655172 | controller | rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-09 19:10:03.655179 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n 2026-03-09 19:10:03.655185 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-09 19:10:03.655190 | controller | username '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 19:05:16 2026-03-09 19:10:03.655194 | controller | | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC 2026-03-09 19:10:03.655198 | controller | transport_url\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ kubectl 2026-03-09 19:10:03.655203 | controller | exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.655207 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ grep transport_url\n 2026-03-09 19:10:03.655211 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ grep -A 5 '\\[oslo_messaging_notifications\\]'\n 2026-03-09 19:10:03.655215 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + notif_transport_url='transport_url 2026-03-09 19:10:03.655220 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-09 19:10:03.655226 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + '[' -z 'transport_url 2026-03-09 19:10:03.655230 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1' 2026-03-09 19:10:03.655235 | controller | ']'\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Found notifications 2026-03-09 19:10:03.655239 | controller | transport_url: transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-09 19:10:03.655251 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Found notifications transport_url: 2026-03-09 19:10:03.655255 | controller | transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1\n 2026-03-09 19:10:03.655260 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n 2026-03-09 19:10:03.655264 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-09 19:10:03.655269 | controller | rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-09 19:10:03.655273 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-09 19:10:03.655277 | controller | vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 2026-03-09 19:10:03.655281 | controller | 19:05:16 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications' 2026-03-09 19:10:03.655285 | controller | in notifications transport_url\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.655290 | controller | | + grep -q watcher-notifications:\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.655294 | controller | | + echo 'transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-09 19:10:03.655298 | controller | \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-09 19:10:03.655302 | controller | username '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 2026-03-09 19:10:03.655307 | controller | 19:05:16 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications' 2026-03-09 19:10:03.655311 | controller | in notifications transport_url\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy 2026-03-09 19:10:03.655315 | controller | | + exit 0\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | test step completed 2026-03-09 19:10:03.655319 | controller | 1-deploy\n logger.go:42: 19:05:16 | watcher-rmquser/2-cleanup-watcher | starting 2026-03-09 19:10:03.655323 | controller | test step 2-cleanup-watcher\n logger.go:42: 19:05:22 | watcher-rmquser/2-cleanup-watcher 2026-03-09 19:10:03.655328 | controller | | test step completed 2-cleanup-watcher\n logger.go:42: 19:05:22 | watcher-rmquser 2026-03-09 19:10:03.655332 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher\n logger.go:42: 2026-03-09 19:10:03.655336 | controller | 19:05:22 | watcher | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-09 19:10:03.655340 | controller | \ logger.go:42: 19:05:22 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher\n 2026-03-09 19:10:03.655345 | controller | \ logger.go:42: 19:05:22 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher\n 2026-03-09 19:10:03.655349 | controller | \ logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | starting test step 2026-03-09 19:10:03.655353 | controller | 1-deploy-with-defaults\n logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655359 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:05:22 2026-03-09 19:10:03.655367 | controller | | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.655373 | controller | \ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.655379 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec 2026-03-09 19:10:03.655385 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.655391 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default 2026-03-09 19:10:03.655397 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-09 19:10:03.655403 | controller | -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-09 19:10:03.655409 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.655415 | controller | -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-09 19:10:03.655422 | controller | get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-09 19:10:03.655428 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-09 19:10:03.655434 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.655440 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-09 19:10:03.655449 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.655456 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.655470 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.655474 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.655479 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.655485 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.655489 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.655493 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.655498 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.655502 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655506 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.655510 | controller | value -c Name -c Type\n logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655515 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:05:24 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655519 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655523 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.655527 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-09 19:10:03.655531 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.655536 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-09 19:10:03.655540 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.655544 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.655548 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default 2026-03-09 19:10:03.655555 | controller | secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep 2026-03-09 19:10:03.655559 | controller | -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.655563 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-09 19:10:03.655567 | controller | == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.655572 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-09 19:10:03.655576 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.655580 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.655584 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.655588 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.655593 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.655597 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.655601 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.655605 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.655610 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.655614 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655618 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.655622 | controller | value -c Name -c Type\n logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655626 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:05:28 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655631 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655635 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.655639 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-09 19:10:03.655643 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.655648 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-09 19:10:03.655652 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.655665 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.655670 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default 2026-03-09 19:10:03.655675 | controller | secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep 2026-03-09 19:10:03.655679 | controller | -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.655683 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-09 19:10:03.655688 | controller | == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.655692 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-09 19:10:03.655696 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.655702 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.655706 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.655711 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.655715 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.655719 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.655724 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.655728 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.655732 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.655736 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655741 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.655745 | controller | value -c Name -c Type\n logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655749 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655754 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655758 | controller | | ++ grep watcher\n logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | 2026-03-09 19:10:03.655762 | controller | ++ awk '{print $1}'\n logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655781 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.655787 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655792 | controller | | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655797 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.655801 | controller | \ logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.655805 | controller | == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655810 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.655814 | controller | \ logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + '[' -n '' ']'\n 2026-03-09 19:10:03.655818 | controller | \ logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | running command: [sh 2026-03-09 19:10:03.655823 | controller | -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.655827 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-09 19:10:03.655831 | controller | \ SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.655835 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n 2026-03-09 19:10:03.655840 | controller | \ [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.655844 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.655848 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default 2026-03-09 19:10:03.655853 | controller | secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep 2026-03-09 19:10:03.655857 | controller | -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.655863 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-09 19:10:03.655875 | controller | == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.655880 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-09 19:10:03.655884 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.655888 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.655893 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.655897 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.655901 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.655905 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.655910 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.655914 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.655921 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.655926 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655931 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.655936 | controller | value -c Name -c Type\n logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655940 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655944 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655948 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655953 | controller | | ++ grep watcher\n logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | 2026-03-09 19:10:03.655957 | controller | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.655961 | controller | value -c Name -c Type -c ID\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655965 | controller | | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655970 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.655974 | controller | \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.655978 | controller | == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.655982 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.655987 | controller | \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.655991 | controller | ']'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d\n 2026-03-09 19:10:03.655995 | controller | \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.656000 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-09 19:10:03.656004 | controller | 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n logger.go:42: 2026-03-09 19:10:03.656010 | controller | 19:05:39 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.656018 | controller | 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:39 2026-03-09 19:10:03.656025 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656031 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656037 | controller | | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 2026-03-09 19:10:03.656042 | controller | 19:05:39 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-09 19:10:03.656047 | controller | 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:39 2026-03-09 19:10:03.656051 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656056 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656060 | controller | | ++ grep -c 'region_name = regionOne'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656064 | controller | | + '[' 9 -ge 1 ']'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656069 | controller | | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-09 19:10:03.656081 | controller | \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n 2026-03-09 19:10:03.656085 | controller | \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + exit 0\n logger.go:42: 2026-03-09 19:10:03.656090 | controller | 19:05:41 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.656094 | controller | \ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.656098 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec 2026-03-09 19:10:03.656103 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.656107 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default 2026-03-09 19:10:03.656111 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-09 19:10:03.656116 | controller | -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-09 19:10:03.656120 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656124 | controller | -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-09 19:10:03.656128 | controller | get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-09 19:10:03.656133 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-09 19:10:03.656137 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656141 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-09 19:10:03.656163 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.656167 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.656172 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.656176 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.656183 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.656187 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.656192 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.656200 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.656205 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.656209 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656213 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.656217 | controller | value -c Name -c Type\n logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656222 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656226 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656230 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656234 | controller | | ++ grep watcher\n logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | 2026-03-09 19:10:03.656239 | controller | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.656243 | controller | value -c Name -c Type -c ID\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656247 | controller | | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656252 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.656256 | controller | \ logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.656260 | controller | == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656264 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.656269 | controller | \ logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.656273 | controller | ']'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d\n 2026-03-09 19:10:03.656277 | controller | \ logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n 2026-03-09 19:10:03.656282 | controller | \ logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.656293 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-09 19:10:03.656298 | controller | 19:05:45 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.656302 | controller | 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:45 2026-03-09 19:10:03.656307 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656311 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656315 | controller | | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 2026-03-09 19:10:03.656320 | controller | 19:05:45 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-09 19:10:03.656324 | controller | 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret 2026-03-09 19:10:03.656328 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-09 19:10:03.656332 | controller | 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:45 2026-03-09 19:10:03.656337 | controller | | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n logger.go:42: 2026-03-09 19:10:03.656341 | controller | 19:05:45 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n logger.go:42: 2026-03-09 19:10:03.656347 | controller | 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators 2026-03-09 19:10:03.656351 | controller | -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:05:46 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656356 | controller | | + '[' '' == '' ']'\n logger.go:42: 19:05:46 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656360 | controller | | + exit 0\n logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | running 2026-03-09 19:10:03.656364 | controller | command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.656369 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-09 19:10:03.656373 | controller | \ SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.656377 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n 2026-03-09 19:10:03.656381 | controller | \ [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.656387 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.656392 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default 2026-03-09 19:10:03.656396 | controller | secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep 2026-03-09 19:10:03.656400 | controller | -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656405 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-09 19:10:03.656409 | controller | == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656413 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-09 19:10:03.656418 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.656422 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.656428 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.656432 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.656436 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.656440 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.656445 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.656449 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.656453 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.656457 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656462 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.656466 | controller | value -c Name -c Type\n logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656470 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656474 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656479 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.656489 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656494 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656500 | controller | | ++ grep watcher\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | 2026-03-09 19:10:03.656504 | controller | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656508 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.656513 | controller | \ logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.656517 | controller | == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656521 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.656525 | controller | \ logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.656530 | controller | ']'\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n 2026-03-09 19:10:03.656534 | controller | \ logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 2026-03-09 19:10:03.656538 | controller | 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret 2026-03-09 19:10:03.656542 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-09 19:10:03.656547 | controller | 19:05:51 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.656551 | controller | 19:05:51 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n 2026-03-09 19:10:03.656555 | controller | \ logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 2026-03-09 19:10:03.656559 | controller | 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret 2026-03-09 19:10:03.656563 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-09 19:10:03.656568 | controller | 19:05:52 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-09 19:10:03.656572 | controller | 19:05:52 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:52 2026-03-09 19:10:03.656576 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656580 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656585 | controller | | ++ grep -c 'region_name = regionOne'\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656589 | controller | | + '[' 9 -ge 1 ']'\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656593 | controller | | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-09 19:10:03.656598 | controller | \ logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n 2026-03-09 19:10:03.656602 | controller | \ logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + exit 0\n logger.go:42: 2026-03-09 19:10:03.656606 | controller | 19:05:53 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.656610 | controller | \ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.656615 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec 2026-03-09 19:10:03.656619 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.656623 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default 2026-03-09 19:10:03.656627 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-09 19:10:03.656632 | controller | -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-09 19:10:03.656638 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656642 | controller | -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-09 19:10:03.656646 | controller | get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-09 19:10:03.656651 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-09 19:10:03.656657 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.656661 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-09 19:10:03.656665 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-09 19:10:03.656676 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.656681 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.656685 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.656689 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.656694 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.656698 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.656702 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.656707 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.656711 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656715 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.656719 | controller | value -c Name -c Type\n logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656724 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656728 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656732 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656737 | controller | | ++ grep watcher\n logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | 2026-03-09 19:10:03.656741 | controller | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.656745 | controller | value -c Name -c Type -c ID\n logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656749 | controller | | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656754 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.656758 | controller | \ logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.656762 | controller | == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656766 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-09 19:10:03.656771 | controller | \ logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.656775 | controller | ']'\n logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ base64 -d\n 2026-03-09 19:10:03.656779 | controller | \ logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n 2026-03-09 19:10:03.656785 | controller | \ logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.656790 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-09 19:10:03.656794 | controller | 19:05:58 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.656798 | controller | 19:05:58 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n 2026-03-09 19:10:03.656804 | controller | \ logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 2026-03-09 19:10:03.656809 | controller | 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret 2026-03-09 19:10:03.656813 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-09 19:10:03.656817 | controller | 19:05:58 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-09 19:10:03.656822 | controller | 19:05:58 | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n 2026-03-09 19:10:03.656826 | controller | \ logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 2026-03-09 19:10:03.656830 | controller | 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret 2026-03-09 19:10:03.656834 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-09 19:10:03.656839 | controller | 19:05:58 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n logger.go:42: 2026-03-09 19:10:03.656843 | controller | 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators 2026-03-09 19:10:03.656847 | controller | -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656852 | controller | | + '[' '' == '' ']'\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults 2026-03-09 19:10:03.656856 | controller | | + exit 0\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | test step 2026-03-09 19:10:03.656867 | controller | completed 1-deploy-with-defaults\n logger.go:42: 19:05:58 | watcher/2-cleanup-watcher 2026-03-09 19:10:03.656872 | controller | | starting test step 2-cleanup-watcher\n logger.go:42: 19:05:58 | watcher/2-cleanup-watcher 2026-03-09 19:10:03.656876 | controller | | test step completed 2-cleanup-watcher\n logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount 2026-03-09 19:10:03.656880 | controller | | starting test step 3-precreate-mariadbaccount\n logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount 2026-03-09 19:10:03.656886 | controller | | MariaDBAccount:watcher-kuttl-default/watcher-precreated created\n logger.go:42: 2026-03-09 19:10:03.656891 | controller | 19:05:58 | watcher/3-precreate-mariadbaccount | test step completed 3-precreate-mariadbaccount\n 2026-03-09 19:10:03.656895 | controller | \ logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | starting 2026-03-09 19:10:03.656899 | controller | test step 4-deploy-with-precreated-account\n logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.656904 | controller | | Secret:watcher-kuttl-default/custom-prometheus-config created\n logger.go:42: 2026-03-09 19:10:03.656908 | controller | 19:05:58 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.656912 | controller | created\n logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.656917 | controller | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.656921 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.656925 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.656929 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.656936 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.656940 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.656945 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.656949 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.656954 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.656958 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.656963 | controller | \ logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.656967 | controller | watcher-kuttl-default\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.656971 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.656976 | controller | \ logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.656980 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.656984 | controller | \ logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.656988 | controller | -1\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.656993 | controller | -v '^$'\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.656997 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657001 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657006 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657012 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.657018 | controller | \ logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | error: Internal 2026-03-09 19:10:03.657027 | controller | error occurred: error executing command in container: container is not created or 2026-03-09 19:10:03.657034 | controller | running\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657039 | controller | ++ echo\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657044 | controller | + '[' 0 == 1 ']'\n logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657048 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657053 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657057 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657070 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657075 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657079 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657083 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657088 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657095 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657099 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657104 | controller | \ logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657108 | controller | watcher-kuttl-default\n logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657112 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657116 | controller | \ logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657121 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657125 | controller | \ logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657131 | controller | -1\n logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657136 | controller | -v '^$'\n logger.go:42: 19:06:01 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657140 | controller | + APIPOD=\n logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657172 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657201 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657206 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657210 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657214 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657219 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657223 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657227 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657232 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657236 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657240 | controller | \ logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657245 | controller | watcher-kuttl-default\n logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657249 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657253 | controller | \ logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657258 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657262 | controller | \ logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657266 | controller | -1\n logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657271 | controller | -v '^$'\n logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657275 | controller | + APIPOD=\n logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657279 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657285 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657290 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657294 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657298 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657311 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657316 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657324 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657328 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657333 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657337 | controller | \ logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657341 | controller | watcher-kuttl-default\n logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657346 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657350 | controller | \ logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657354 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657359 | controller | \ logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657363 | controller | -1\n logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657367 | controller | -v '^$'\n logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657371 | controller | + APIPOD=\n logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657376 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657380 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657386 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657390 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657395 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657399 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657403 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657408 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657412 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657416 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657421 | controller | \ logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657425 | controller | watcher-kuttl-default\n logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657429 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657435 | controller | \ logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657440 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657444 | controller | \ logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657448 | controller | -1\n logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657453 | controller | -v '^$'\n logger.go:42: 19:06:05 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657457 | controller | + APIPOD=\n logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657461 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657466 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657470 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657474 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657478 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657483 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657487 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657499 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657504 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657508 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657512 | controller | \ logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657517 | controller | watcher-kuttl-default\n logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657521 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657525 | controller | \ logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657530 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657534 | controller | \ logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657541 | controller | -1\n logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657545 | controller | -v '^$'\n logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657549 | controller | + APIPOD=\n logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657554 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657558 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657562 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657567 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657571 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657575 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657579 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657586 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657590 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657594 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657599 | controller | \ logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657605 | controller | watcher-kuttl-default\n logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657609 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657615 | controller | \ logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657619 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657624 | controller | \ logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657628 | controller | -1\n logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657632 | controller | -v '^$'\n logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657637 | controller | + APIPOD=\n logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657641 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657645 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657650 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657654 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657658 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657662 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657667 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657678 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657684 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657688 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657692 | controller | \ logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657697 | controller | watcher-kuttl-default\n logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657701 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657705 | controller | \ logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657710 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657714 | controller | \ logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657718 | controller | -1\n logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657722 | controller | -v '^$'\n logger.go:42: 19:06:09 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657729 | controller | + APIPOD=\n logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657733 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657737 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657742 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657746 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657750 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657755 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657759 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657763 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657768 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657772 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657776 | controller | \ logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657781 | controller | watcher-kuttl-default\n logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657785 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657789 | controller | \ logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657793 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657798 | controller | \ logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657802 | controller | -1\n logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657806 | controller | -v '^$'\n logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657813 | controller | + APIPOD=\n logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657817 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657821 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657826 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657830 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657834 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657839 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657843 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657847 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657852 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657863 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657867 | controller | \ logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657874 | controller | watcher-kuttl-default\n logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657878 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657882 | controller | \ logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657887 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657891 | controller | \ logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.657895 | controller | -1\n logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657899 | controller | -v '^$'\n logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657904 | controller | + APIPOD=\n logger.go:42: 19:06:12 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657908 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657912 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657917 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.657921 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.657925 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.657930 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.657934 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.657938 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.657943 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.657947 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.657951 | controller | \ logger.go:42: 19:06:12 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.657955 | controller | watcher-kuttl-default\n logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657960 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.657964 | controller | \ logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.657968 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.657972 | controller | \ logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.657977 | controller | -v '^$'\n logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.657981 | controller | ++ head -1\n logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657985 | controller | | + APIPOD=\n logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.657990 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.657994 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.657998 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.658003 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.658008 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.658019 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.658025 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.658032 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.658038 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.658043 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.658047 | controller | \ logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.658059 | controller | watcher-kuttl-default\n logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658064 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.658068 | controller | \ logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.658073 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.658077 | controller | \ logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.658081 | controller | -v '^$'\n logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.658085 | controller | ++ head -1\n logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658090 | controller | | + APIPOD=\n logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658094 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-09 19:10:03.658098 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-09 19:10:03.658102 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-09 19:10:03.658107 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.658111 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.658115 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.658119 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.658124 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.658128 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.658132 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.658136 | controller | \ logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.658141 | controller | watcher-kuttl-default\n logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658158 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.658165 | controller | \ logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.658169 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.658173 | controller | \ logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.658180 | controller | -1\n logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.658184 | controller | -v '^$'\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.658188 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658193 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658197 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658201 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.658206 | controller | \ logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | error: unable 2026-03-09 19:10:03.658213 | controller | to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-09 19:10:03.658217 | controller | 19:06:16 | watcher/4-deploy-with-precreated-account | ++ echo\n logger.go:42: 2026-03-09 19:10:03.658221 | controller | 19:06:16 | watcher/4-deploy-with-precreated-account | + '[' 0 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.658226 | controller | 19:06:17 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.658230 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.658234 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.658238 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.658242 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.658247 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.658251 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.658255 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.658267 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.658272 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.658276 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.658282 | controller | \ logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.658287 | controller | watcher-kuttl-default\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658291 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.658295 | controller | \ logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.658299 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.658304 | controller | \ logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.658308 | controller | -1\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.658312 | controller | -v '^$'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.658319 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658324 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658328 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658334 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.658338 | controller | \ logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.658343 | controller | '#' Global config\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658347 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658351 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658355 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.658360 | controller | \ logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.658364 | controller | '#' Service config\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658368 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658372 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.658377 | controller | \ logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.658381 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.658385 | controller | \ logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.658390 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.658394 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.658398 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.658403 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.658407 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.658411 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.658416 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.658420 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.658424 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.658429 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.658439 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.658443 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.658447 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.658452 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.658456 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.658460 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.658465 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.658471 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.658475 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.658479 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.658483 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.658488 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.658492 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.658496 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.658500 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.658505 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.658509 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.658513 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.658517 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.658522 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.658526 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.658530 | controller | period = 900\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658534 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658539 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658543 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.658547 | controller | \ logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.658551 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.658562 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.658566 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.658570 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.658575 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.658579 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.658583 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.658587 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.658591 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.658596 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.658600 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.658604 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.658608 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.658614 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.658619 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.658623 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.658633 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.658638 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.658642 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.658647 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.658652 | controller | 19:06:18 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.658657 | controller | 19:06:19 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.658661 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.658665 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.658670 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.658674 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.658678 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.658682 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.658686 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.658691 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.658695 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.658699 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.658704 | controller | \ logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.658708 | controller | watcher-kuttl-default\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658712 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.658716 | controller | \ logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.658720 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.658725 | controller | \ logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.658729 | controller | -1\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.658733 | controller | -v '^$'\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.658737 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658742 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658746 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658750 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.658754 | controller | \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.658760 | controller | '#' Global config\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658765 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658769 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658773 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.658777 | controller | \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.658782 | controller | '#' Service config\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658786 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658790 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.658794 | controller | \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.658799 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.658803 | controller | \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.658812 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.658817 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.658821 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.658826 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.658830 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.658834 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.658839 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.658843 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.658849 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.658853 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.658857 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.658862 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.658866 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.658870 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.658874 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.658879 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.658883 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.658887 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.658891 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.658897 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.658902 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.658906 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.658910 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.658914 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.658918 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.658923 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.658927 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.658931 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.658935 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.658940 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.658944 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.658948 | controller | period = 900\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658952 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658957 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.658961 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.658965 | controller | \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.658969 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.658978 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.658982 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.658991 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.658996 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.659000 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.659005 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.659011 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.659020 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.659026 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.659032 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.659037 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.659042 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.659046 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.659050 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.659057 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.659061 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.659066 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.659070 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.659074 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.659078 | controller | 19:06:20 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.659083 | controller | 19:06:21 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.659087 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.659091 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.659097 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.659102 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.659106 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.659110 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.659114 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.659119 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.659123 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.659127 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.659132 | controller | \ logger.go:42: 19:06:21 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.659136 | controller | watcher-kuttl-default\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659140 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.659157 | controller | \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.659164 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.659168 | controller | \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.659172 | controller | -1\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.659177 | controller | -v '^$'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.659181 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659185 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659189 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.659194 | controller | \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.659198 | controller | -c '^# Global config'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659210 | controller | | ++ echo '#' Global config\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659217 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659221 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659226 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.659230 | controller | \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.659234 | controller | '#' Service config\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659239 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659243 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.659247 | controller | \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.659251 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.659256 | controller | \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.659260 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.659264 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.659268 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.659273 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.659277 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.659281 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.659286 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.659290 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.659294 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.659298 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.659302 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.659307 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.659311 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.659315 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.659319 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.659324 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.659328 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.659332 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.659336 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.659340 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.659345 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.659351 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.659355 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.659359 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.659364 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.659368 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.659372 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.659376 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.659391 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.659403 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.659407 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.659412 | controller | period = 900\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659419 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659423 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659428 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.659432 | controller | \ logger.go:42: 19:06:23 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.659436 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.659445 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.659449 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.659453 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.659457 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.659462 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.659468 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.659474 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.659479 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.659485 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.659491 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.659496 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.659502 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.659508 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.659514 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.659520 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.659526 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.659535 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.659542 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.659548 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.659553 | controller | 19:06:23 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.659559 | controller | 19:06:24 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.659564 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.659570 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.659575 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.659581 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.659586 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.659592 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.659597 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.659607 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.659613 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.659619 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.659624 | controller | \ logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.659642 | controller | watcher-kuttl-default\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659648 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.659655 | controller | \ logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.659659 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.659663 | controller | \ logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.659667 | controller | -1\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.659672 | controller | -v '^$'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.659676 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659680 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659684 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659689 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.659693 | controller | \ logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.659697 | controller | '#' Global config\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659701 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659705 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659712 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.659717 | controller | \ logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.659721 | controller | '#' Service config\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659725 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659729 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.659736 | controller | \ logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.659740 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.659744 | controller | \ logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.659749 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.659753 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.659757 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.659790 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.659796 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.659800 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.659805 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.659809 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.659813 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.659817 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.659822 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.659826 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.659832 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.659836 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.659841 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.659845 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.659857 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.659862 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.659866 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.659870 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.659874 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.659879 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.659883 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.659889 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.659894 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.659898 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.659902 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.659906 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.659911 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.659915 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.659919 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.659923 | controller | period = 900\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659928 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659932 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.659936 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.659940 | controller | \ logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.659944 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.659953 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.659957 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.659961 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.659965 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.659970 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.659974 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.659978 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.659982 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.659987 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.659991 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.659995 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.659999 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.660004 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.660009 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.660015 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.660024 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.660030 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.660039 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.660043 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.660048 | controller | 19:06:25 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.660052 | controller | 19:06:26 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.660064 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.660069 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.660073 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.660077 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.660081 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.660086 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.660090 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.660094 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.660099 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.660103 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.660107 | controller | \ logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.660111 | controller | watcher-kuttl-default\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660116 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.660120 | controller | \ logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.660124 | controller | -v '^$'\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.660128 | controller | ++ head -1\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660132 | controller | | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.660137 | controller | \ logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n 2026-03-09 19:10:03.660141 | controller | \ logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + '[' -n 2026-03-09 19:10:03.660162 | controller | watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660167 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660171 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.660175 | controller | \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.660180 | controller | '#' Global config\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660184 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660188 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660195 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.660199 | controller | \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.660207 | controller | '#' Service config\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660211 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660216 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.660220 | controller | \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.660224 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.660228 | controller | \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.660233 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.660237 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.660241 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.660246 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.660257 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.660262 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.660266 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.660271 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.660275 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.660279 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.660284 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.660288 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.660292 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.660296 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.660301 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.660305 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.660309 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.660313 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.660318 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.660322 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.660326 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.660330 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.660337 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.660341 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.660347 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.660351 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.660356 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.660360 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.660364 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.660368 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.660373 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.660377 | controller | period = 900\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660381 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660385 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660390 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.660394 | controller | \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.660398 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.660407 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.660411 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.660415 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.660419 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.660424 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.660428 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.660432 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.660436 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.660447 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.660452 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.660456 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.660460 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.660465 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.660469 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.660473 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.660478 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.660482 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.660486 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.660490 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.660496 | controller | 19:06:27 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.660501 | controller | 19:06:28 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.660505 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.660509 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.660513 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.660518 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.660522 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.660526 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.660530 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.660535 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.660539 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.660543 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.660550 | controller | \ logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.660554 | controller | watcher-kuttl-default\n logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660558 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.660562 | controller | \ logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.660567 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.660571 | controller | \ logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.660575 | controller | -1\n logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.660580 | controller | -v '^$'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.660587 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660592 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660597 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660601 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.660605 | controller | \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.660609 | controller | '#' Global config\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660614 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660618 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660622 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.660626 | controller | \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.660637 | controller | '#' Service config\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660642 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660648 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.660652 | controller | \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.660657 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.660661 | controller | \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.660667 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.660671 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.660676 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.660680 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.660684 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.660689 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.660693 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.660697 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.660701 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.660706 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.660710 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.660714 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.660718 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.660722 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.660727 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.660731 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.660735 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.660739 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.660744 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.660748 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.660752 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.660756 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.660761 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.660765 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.660769 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.660773 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.660779 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.660784 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.660788 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.660792 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.660796 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.660801 | controller | period = 900\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660805 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660815 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660819 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.660826 | controller | \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.660831 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.660839 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.660843 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.660848 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.660852 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.660856 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.660860 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.660865 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.660869 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.660873 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.660877 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.660883 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.660888 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.660892 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.660896 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.660900 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.660905 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.660909 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.660913 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.660917 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.660923 | controller | 19:06:29 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.660928 | controller | 19:06:31 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.660934 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.660938 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.660942 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.660946 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.660951 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.660955 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.660959 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.660963 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.660968 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.660972 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.660976 | controller | \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.660980 | controller | watcher-kuttl-default\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.660985 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.660989 | controller | \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.660993 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.660997 | controller | \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.661002 | controller | -1\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.661015 | controller | -v '^$'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.661022 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661028 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661034 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661040 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.661045 | controller | \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661051 | controller | '#' Global config\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661056 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661061 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661067 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.661073 | controller | \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661078 | controller | '#' Service config\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661084 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661089 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.661098 | controller | \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.661104 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.661109 | controller | \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661114 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.661123 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.661128 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.661134 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.661139 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.661159 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.661168 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.661174 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.661179 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.661185 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.661191 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.661196 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.661202 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.661207 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.661213 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.661219 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.661225 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.661231 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.661236 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.661242 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.661248 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.661253 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.661270 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.661276 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.661282 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.661287 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.661291 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.661296 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.661303 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.661307 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.661311 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.661316 | controller | period = 900\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661320 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661324 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661328 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.661333 | controller | \ logger.go:42: 19:06:32 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661337 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.661345 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.661350 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.661354 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.661358 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.661362 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.661366 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.661371 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.661377 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.661381 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.661386 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.661390 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.661394 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.661398 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.661402 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.661407 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.661411 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.661415 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.661419 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.661423 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.661428 | controller | 19:06:32 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.661432 | controller | 19:06:33 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.661436 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.661440 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.661446 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.661451 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.661455 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.661459 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.661471 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.661476 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.661480 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.661485 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.661491 | controller | \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.661495 | controller | watcher-kuttl-default\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661499 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.661504 | controller | \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.661508 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.661512 | controller | \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.661516 | controller | -1\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.661520 | controller | -v '^$'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.661525 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661529 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661533 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661537 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.661542 | controller | \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661546 | controller | '#' Global config\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661550 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661555 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661559 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.661563 | controller | \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661568 | controller | '#' Service config\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661572 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661579 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.661584 | controller | \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.661590 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.661594 | controller | \ logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661599 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.661603 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.661607 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.661611 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.661616 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.661620 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.661624 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.661629 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.661633 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.661644 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.661649 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.661653 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.661657 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.661662 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.661666 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.661670 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.661675 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.661679 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.661683 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.661688 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.661692 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.661696 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.661701 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.661705 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.661709 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.661713 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.661718 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.661722 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.661726 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.661731 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.661737 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.661741 | controller | period = 900\n logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661746 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661752 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661756 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.661761 | controller | \ logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661765 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.661775 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.661779 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.661783 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.661788 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.661792 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.661796 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.661801 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.661805 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.661809 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.661814 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.661818 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.661822 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.661826 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.661831 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.661842 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.661846 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.661851 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.661855 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.661859 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.661864 | controller | 19:06:34 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.661868 | controller | 19:06:35 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.661872 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.661877 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.661881 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.661887 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.661891 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.661896 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.661900 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.661904 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.661909 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.661913 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.661917 | controller | \ logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.661921 | controller | watcher-kuttl-default\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661926 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.661930 | controller | \ logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.661934 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.661939 | controller | \ logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.661943 | controller | -1\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.661947 | controller | -v '^$'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.661951 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661956 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661960 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661964 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.661969 | controller | \ logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661973 | controller | '#' Global config\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661977 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661982 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661986 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.661990 | controller | \ logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.661994 | controller | '#' Service config\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.661999 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662003 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.662009 | controller | \ logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.662015 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.662030 | controller | \ logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662040 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.662046 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.662051 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.662055 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.662060 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.662064 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.662068 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.662073 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.662079 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.662084 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.662088 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.662092 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.662097 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.662102 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.662107 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.662111 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.662115 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.662120 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.662124 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.662128 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.662133 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.662137 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.662141 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.662162 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.662167 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.662172 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.662176 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.662180 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.662185 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.662189 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.662195 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.662202 | controller | period = 900\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662207 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662211 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662215 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.662220 | controller | \ logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662224 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.662232 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.662245 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.662250 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.662254 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.662259 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.662263 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.662267 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.662272 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.662276 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.662280 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.662285 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.662289 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.662293 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.662298 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.662302 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.662306 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.662311 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.662316 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.662322 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.662328 | controller | 19:06:36 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.662334 | controller | 19:06:37 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.662340 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.662346 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.662354 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.662360 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.662366 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.662374 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.662380 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.662386 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.662393 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.662399 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.662405 | controller | \ logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.662411 | controller | watcher-kuttl-default\n logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662417 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.662423 | controller | \ logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.662429 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.662434 | controller | \ logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.662440 | controller | -1\n logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.662445 | controller | -v '^$'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.662450 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662454 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662458 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662462 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.662476 | controller | \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662481 | controller | '#' Global config\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662485 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662489 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662494 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.662498 | controller | \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662502 | controller | '#' Service config\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662506 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662511 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.662515 | controller | \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.662519 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.662523 | controller | \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662528 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.662537 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.662542 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.662546 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.662550 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.662555 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.662559 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.662563 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.662567 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.662575 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.662579 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.662583 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.662587 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.662592 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.662596 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.662600 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.662604 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.662609 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.662613 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.662617 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.662621 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.662626 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.662630 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.662634 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.662638 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.662643 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.662647 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.662653 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.662664 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.662668 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.662673 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.662677 | controller | period = 900\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662681 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662687 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662692 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.662696 | controller | \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662700 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.662709 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.662713 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.662717 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.662721 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.662725 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.662730 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.662734 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.662738 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.662742 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.662747 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.662751 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.662755 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.662759 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.662764 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.662768 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.662772 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.662776 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.662780 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.662785 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.662789 | controller | 19:06:38 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.662793 | controller | 19:06:40 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.662797 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.662802 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.662806 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.662810 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.662814 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.662818 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.662823 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.662829 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.662833 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.662838 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.662850 | controller | \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.662855 | controller | watcher-kuttl-default\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662859 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.662864 | controller | \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.662868 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.662872 | controller | \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.662877 | controller | -1\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.662881 | controller | -v '^$'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.662885 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662889 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662894 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662898 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.662902 | controller | \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662906 | controller | '#' Global config\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662912 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662917 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662921 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.662926 | controller | \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662931 | controller | '#' Service config\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662935 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.662939 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.662944 | controller | \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.662948 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.662952 | controller | \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.662956 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.662961 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.662965 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.662971 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.662975 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.662980 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.662984 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.662988 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.662992 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.662997 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.663001 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.663006 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.663012 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.663017 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.663023 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.663038 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.663045 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.663049 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.663054 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.663058 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.663062 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.663066 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.663071 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.663075 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.663079 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.663083 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.663087 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.663091 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.663096 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.663100 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.663104 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.663108 | controller | period = 900\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663113 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663117 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663123 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.663127 | controller | \ logger.go:42: 19:06:41 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.663132 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.663140 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.663158 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.663165 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.663169 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.663173 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.663178 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.663182 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.663186 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.663190 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.663195 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.663199 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.663203 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.663207 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.663212 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.663218 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.663222 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.663227 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.663231 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.663235 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.663239 | controller | 19:06:41 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.663252 | controller | 19:06:42 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.663256 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.663261 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.663265 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.663269 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.663273 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.663278 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.663282 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.663286 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.663292 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.663297 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.663301 | controller | \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.663307 | controller | watcher-kuttl-default\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663311 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.663315 | controller | \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.663319 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.663324 | controller | \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.663328 | controller | -1\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.663332 | controller | -v '^$'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.663336 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663341 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663345 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663349 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.663353 | controller | \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.663357 | controller | '#' Global config\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663362 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663366 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663370 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.663374 | controller | \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.663379 | controller | '#' Service config\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663383 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663387 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.663391 | controller | \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.663396 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.663400 | controller | \ logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.663404 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.663408 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.663413 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.663424 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.663431 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.663435 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.663439 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.663444 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.663448 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.663452 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.663456 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.663460 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.663465 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.663469 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.663473 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.663477 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.663482 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.663486 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.663490 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.663496 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.663500 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.663505 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.663509 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.663513 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.663517 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.663522 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.663526 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.663530 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.663534 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.663539 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.663543 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.663547 | controller | period = 900\n logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663551 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663556 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663560 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.663564 | controller | \ logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.663570 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.663579 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.663583 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.663587 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.663591 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.663595 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.663600 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.663604 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.663614 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.663619 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.663623 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.663627 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.663632 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.663636 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.663640 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.663644 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.663648 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.663653 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.663660 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.663664 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.663669 | controller | 19:06:43 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.663673 | controller | 19:06:44 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.663677 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.663681 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.663686 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.663690 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.663694 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.663698 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.663703 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.663707 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.663711 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.663717 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.663722 | controller | \ logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.663726 | controller | watcher-kuttl-default\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663730 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.663734 | controller | \ logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.663739 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.663745 | controller | \ logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.663749 | controller | -v '^$'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.663753 | controller | ++ head -1\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663770 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663775 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663780 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663784 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.663788 | controller | \ logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.663792 | controller | '#' Global config\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663797 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663801 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663805 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.663816 | controller | \ logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.663820 | controller | '#' Service config\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663825 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663829 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.663833 | controller | \ logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.663837 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.663842 | controller | \ logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.663846 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.663850 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.663854 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.663859 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.663863 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.663867 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.663873 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.663878 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.663882 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.663886 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.663890 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.663894 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.663899 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.663903 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.663907 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.663911 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.663916 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.663920 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.663924 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.663928 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.663932 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.663937 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.663941 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.663945 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.663949 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.663953 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.663958 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.663962 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.663966 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.663970 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.663975 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.663979 | controller | period = 900\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663983 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663994 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.663998 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.664002 | controller | \ logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664008 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.664023 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.664029 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.664035 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.664042 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.664048 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.664052 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.664056 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.664060 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.664064 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.664069 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.664073 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.664077 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.664081 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.664086 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.664090 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.664094 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.664098 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.664102 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.664107 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.664111 | controller | 19:06:45 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.664115 | controller | 19:06:47 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.664119 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.664123 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.664128 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.664132 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.664136 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.664140 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.664158 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.664165 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.664169 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.664173 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.664178 | controller | \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.664184 | controller | watcher-kuttl-default\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664188 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.664192 | controller | \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.664197 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.664201 | controller | \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.664214 | controller | -1\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.664218 | controller | -v '^$'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.664223 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664227 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664231 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664236 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.664240 | controller | \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664244 | controller | '#' Global config\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664248 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664252 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664257 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.664261 | controller | \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664265 | controller | '#' Service config\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664269 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664274 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.664278 | controller | \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.664282 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.664286 | controller | \ logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664290 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.664295 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.664299 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.664303 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.664307 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.664312 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.664316 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.664323 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.664327 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.664331 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.664336 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.664342 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.664346 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.664350 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.664354 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.664358 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.664363 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.664367 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.664371 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.664375 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.664379 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.664391 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.664395 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.664399 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.664404 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.664408 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.664414 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.664418 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.664422 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.664427 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.664433 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.664437 | controller | period = 900\n logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664441 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664445 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664450 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.664454 | controller | \ logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664458 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.664466 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.664471 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.664477 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.664481 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.664485 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.664490 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.664494 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.664498 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.664502 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.664507 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.664512 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.664518 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.664524 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.664529 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.664534 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.664540 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.664545 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.664550 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.664556 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.664561 | controller | 19:06:48 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.664567 | controller | 19:06:49 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.664573 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.664578 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.664584 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.664589 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.664595 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.664607 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.664612 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.664616 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.664621 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.664625 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.664629 | controller | \ logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.664634 | controller | watcher-kuttl-default\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664638 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.664644 | controller | \ logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.664649 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.664653 | controller | \ logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.664657 | controller | -1\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.664661 | controller | -v '^$'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.664666 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664670 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664674 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664678 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.664683 | controller | \ logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664687 | controller | '#' Global config\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664691 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664695 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664699 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.664704 | controller | \ logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664708 | controller | '#' Service config\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664712 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664716 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.664721 | controller | \ logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.664725 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.664729 | controller | \ logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664733 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.664738 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.664742 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.664748 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.664753 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.664757 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.664761 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.664768 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.664779 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.664786 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.664790 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.664795 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.664799 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.664803 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.664807 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.664812 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.664816 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.664820 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.664824 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.664828 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.664833 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.664837 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.664841 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.664845 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.664850 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.664854 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.664858 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.664862 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.664866 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.664871 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.664875 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.664879 | controller | period = 900\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664883 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664888 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.664892 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.664898 | controller | \ logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.664902 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.664911 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.664915 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.664919 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.664923 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.664932 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.664937 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.664941 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.664945 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.664949 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.664954 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.664958 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.664962 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.664966 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.664977 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.664981 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.664986 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.664990 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.664994 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.665000 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.665005 | controller | 19:06:50 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.665011 | controller | 19:06:51 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.665017 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.665023 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.665029 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.665035 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.665040 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.665045 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.665049 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.665053 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.665058 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.665062 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.665068 | controller | \ logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.665072 | controller | watcher-kuttl-default\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665077 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.665081 | controller | \ logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.665085 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.665091 | controller | \ logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.665095 | controller | -1\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.665100 | controller | -v '^$'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.665106 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665112 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665118 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665124 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.665130 | controller | \ logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.665135 | controller | '#' Global config\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665139 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665156 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665165 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.665169 | controller | \ logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.665176 | controller | '#' Service config\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665180 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665184 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.665189 | controller | \ logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.665202 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.665208 | controller | \ logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.665212 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.665216 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.665221 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.665225 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.665230 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.665234 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.665238 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.665242 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.665247 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.665251 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.665255 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.665261 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.665266 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.665270 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.665274 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.665278 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.665283 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.665287 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.665291 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.665295 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.665299 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.665304 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.665308 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.665312 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.665316 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.665321 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.665325 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.665329 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.665333 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.665338 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.665342 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.665346 | controller | period = 900\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665350 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665355 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665359 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.665363 | controller | \ logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.665367 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.665376 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.665387 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.665392 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.665396 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.665400 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.665407 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.665411 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.665415 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.665420 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.665424 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.665428 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.665432 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.665436 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.665441 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.665445 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.665449 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.665453 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.665458 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.665462 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.665466 | controller | 19:06:52 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.665472 | controller | 19:06:53 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.665476 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.665481 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.665485 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.665489 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.665493 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.665498 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.665502 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.665508 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.665512 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.665516 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.665521 | controller | \ logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.665525 | controller | watcher-kuttl-default\n logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665529 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.665533 | controller | \ logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.665538 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.665542 | controller | \ logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.665548 | controller | -1\n logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.665552 | controller | -v '^$'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.665556 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665560 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665566 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665577 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.665581 | controller | \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.665586 | controller | '#' Global config\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665590 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665594 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665599 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.665603 | controller | \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.665607 | controller | '#' Service config\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665611 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665616 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.665620 | controller | \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.665624 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.665628 | controller | \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.665633 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.665637 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.665641 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.665645 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.665650 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.665654 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.665658 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.665662 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.665666 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.665671 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.665675 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.665679 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.665687 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.665691 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.665695 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.665699 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.665704 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.665708 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.665712 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.665716 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.665721 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.665725 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.665729 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.665735 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.665739 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.665743 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.665748 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.665758 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.665763 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.665767 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.665772 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.665776 | controller | period = 900\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665780 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665784 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665788 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.665793 | controller | \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.665797 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.665805 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.665810 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.665814 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.665818 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.665822 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.665826 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.665831 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.665837 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.665841 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.665845 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.665849 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.665853 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.665858 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.665862 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.665866 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.665870 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.665875 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.665879 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.665883 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.665887 | controller | 19:06:54 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.665891 | controller | 19:06:56 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.665896 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.665902 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.665906 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.665910 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.665914 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.665919 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.665923 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.665927 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.665931 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.665942 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.665947 | controller | \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.665951 | controller | watcher-kuttl-default\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665955 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.665959 | controller | \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.665964 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.665968 | controller | \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.665972 | controller | -1\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.665976 | controller | -v '^$'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.665982 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665987 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665991 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.665995 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.666001 | controller | \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666006 | controller | '#' Global config\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666012 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666018 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666024 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.666030 | controller | \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666036 | controller | '#' Service config\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666041 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666046 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.666050 | controller | \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.666054 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.666058 | controller | \ logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666063 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.666067 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.666071 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.666075 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.666080 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.666084 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.666088 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.666092 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.666096 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.666101 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.666105 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.666109 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.666113 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.666118 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.666132 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.666136 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.666141 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.666156 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.666163 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.666167 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.666172 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.666176 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.666180 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.666184 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.666188 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.666193 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.666197 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.666201 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.666205 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.666210 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.666214 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.666218 | controller | period = 900\n logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666222 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666227 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666231 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.666235 | controller | \ logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666239 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.666248 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.666252 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.666256 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.666260 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.666265 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.666269 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.666273 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.666277 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.666282 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.666290 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.666295 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.666301 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.666305 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.666310 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.666314 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.666318 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.666322 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.666327 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.666331 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.666342 | controller | 19:06:57 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.666347 | controller | 19:06:58 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.666351 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.666355 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.666360 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.666364 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.666368 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.666372 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.666376 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.666381 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.666385 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.666389 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.666394 | controller | \ logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.666398 | controller | watcher-kuttl-default\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666402 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.666406 | controller | \ logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.666410 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.666415 | controller | \ logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.666419 | controller | -1\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.666423 | controller | -v '^$'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.666427 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666432 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666438 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666442 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.666446 | controller | \ logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666450 | controller | '#' Global config\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666455 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666459 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666463 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.666467 | controller | \ logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666472 | controller | '#' Service config\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666476 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666480 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.666484 | controller | \ logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.666489 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.666493 | controller | \ logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666497 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.666501 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.666505 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.666515 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.666519 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.666524 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.666528 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.666532 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.666536 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.666541 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.666545 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.666549 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.666553 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.666558 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.666562 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.666566 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.666574 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.666578 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.666582 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.666587 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.666591 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.666595 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.666599 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.666604 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.666608 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.666612 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.666616 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.666621 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.666625 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.666629 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.666633 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.666637 | controller | period = 900\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666642 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666646 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666652 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.666656 | controller | \ logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666661 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.666669 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.666673 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.666678 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.666682 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.666686 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.666690 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.666695 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.666704 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.666708 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.666713 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.666719 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.666723 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.666727 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.666731 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.666736 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.666740 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.666744 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.666748 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.666752 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.666757 | controller | 19:06:59 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.666761 | controller | 19:07:00 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.666765 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.666769 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.666774 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.666778 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.666782 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.666786 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.666790 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.666795 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.666799 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.666803 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.666808 | controller | \ logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.666812 | controller | watcher-kuttl-default\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666816 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.666823 | controller | \ logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.666827 | controller | -1\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.666832 | controller | -v '^$'\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.666836 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.666840 | controller | \ logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n 2026-03-09 19:10:03.666844 | controller | \ logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + '[' -n 2026-03-09 19:10:03.666849 | controller | watcher-kuttl-api-0 ']'\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666853 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666859 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.666863 | controller | \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666867 | controller | '#' Global config\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666872 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666876 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666880 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.666889 | controller | \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666894 | controller | '#' Service config\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666898 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.666902 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.666907 | controller | \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.666911 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.666915 | controller | \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.666919 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.666924 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.666928 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.666932 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.666936 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.666941 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.666945 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.666949 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.666953 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.666957 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.666962 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.666966 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.666972 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.666976 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.666980 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.666985 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.666989 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.666993 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.666999 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.667003 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.667009 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.667015 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.667021 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.667028 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.667033 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.667038 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.667043 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.667047 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.667051 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.667055 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.667060 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.667064 | controller | period = 900\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667074 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667079 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667086 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.667090 | controller | \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.667094 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.667103 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.667107 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.667111 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.667116 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.667120 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.667124 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.667128 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.667134 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.667139 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.667143 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.667174 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.667179 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.667185 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.667190 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.667194 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.667198 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.667202 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.667207 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.667211 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.667215 | controller | 19:07:01 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.667219 | controller | 19:07:03 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.667224 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.667228 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.667232 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.667236 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.667241 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.667245 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.667249 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.667253 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.667258 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.667262 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.667266 | controller | \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.667271 | controller | watcher-kuttl-default\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667275 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.667279 | controller | \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.667283 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.667295 | controller | \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.667300 | controller | -1\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.667304 | controller | -v '^$'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.667308 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667313 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667317 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667321 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.667326 | controller | \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.667332 | controller | '#' Global config\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667336 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667340 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667345 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.667349 | controller | \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.667353 | controller | '#' Service config\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667358 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667364 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.667368 | controller | \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.667373 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.667377 | controller | \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.667381 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.667385 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.667390 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.667394 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.667398 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.667403 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.667407 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.667413 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.667419 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.667425 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.667435 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.667441 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.667447 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.667452 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.667459 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.667463 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.667468 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.667472 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.667476 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.667482 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.667495 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.667500 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.667504 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.667508 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.667513 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.667517 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.667521 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.667525 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.667530 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.667534 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.667538 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.667543 | controller | period = 900\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667547 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667551 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667555 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.667560 | controller | \ logger.go:42: 19:07:04 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.667564 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.667572 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.667576 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.667581 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.667585 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.667589 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.667593 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.667598 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.667602 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.667606 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.667610 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.667614 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.667619 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.667623 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.667627 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.667633 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.667637 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.667642 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.667646 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.667650 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.667654 | controller | 19:07:04 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.667659 | controller | 19:07:05 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.667663 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.667667 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.667671 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.667676 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.667688 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.667692 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.667697 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.667701 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.667706 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.667710 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.667714 | controller | \ logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.667718 | controller | watcher-kuttl-default\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667722 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.667727 | controller | \ logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.667731 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.667735 | controller | \ logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.667739 | controller | -1\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.667744 | controller | -v '^$'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.667748 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667753 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667827 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667838 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.667845 | controller | \ logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.667850 | controller | '#' Global config\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667855 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667861 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667865 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.667869 | controller | \ logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.667874 | controller | '#' Service config\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667878 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.667882 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.667887 | controller | \ logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.667891 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.667895 | controller | \ logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.667900 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.667912 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.667916 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.667921 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.667925 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.667929 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.667933 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.667946 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.667951 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.667955 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.667960 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.667964 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.667968 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.667973 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.667977 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.667981 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.667985 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.667989 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.667994 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.667998 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.668002 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.668010 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.668016 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.668022 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.668028 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.668034 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.668040 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.668046 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.668050 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.668054 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.668058 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.668063 | controller | period = 900\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668067 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668071 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668075 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.668080 | controller | \ logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.668084 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.668092 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.668097 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.668101 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.668105 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.668109 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.668113 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.668118 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.668122 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.668126 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.668130 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.668135 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.668139 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.668163 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.668171 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.668176 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.668180 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.668187 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.668191 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.668195 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.668199 | controller | 19:07:06 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.668204 | controller | 19:07:07 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-09 19:10:03.668208 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-09 19:10:03.668212 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-09 19:10:03.668216 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-09 19:10:03.668221 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.668225 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-09 19:10:03.668229 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-09 19:10:03.668233 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-09 19:10:03.668238 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-09 19:10:03.668242 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.668249 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-09 19:10:03.668254 | controller | \ logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.668258 | controller | watcher-kuttl-default\n logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668263 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-09 19:10:03.668267 | controller | \ logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.668271 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-09 19:10:03.668275 | controller | \ logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.668279 | controller | -1\n logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.668284 | controller | -v '^$'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | 2026-03-09 19:10:03.668288 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668292 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668296 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668300 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-09 19:10:03.668305 | controller | \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.668309 | controller | '#' Global config\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668313 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668317 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668323 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-09 19:10:03.668328 | controller | \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.668334 | controller | '#' Service config\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668340 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668348 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-09 19:10:03.668364 | controller | \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-09 19:10:03.668373 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-09 19:10:03.668379 | controller | \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.668385 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-09 19:10:03.668390 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.668395 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.668400 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-09 19:10:03.668404 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-09 19:10:03.668408 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.668412 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.668417 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.668421 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.668425 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.668429 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.668433 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.668438 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.668442 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-09 19:10:03.668446 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-09 19:10:03.668450 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-09 19:10:03.668454 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-09 19:10:03.668459 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-09 19:10:03.668463 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.668467 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.668471 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.668476 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-09 19:10:03.668480 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.668486 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-09 19:10:03.668490 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-09 19:10:03.668495 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.668499 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-09 19:10:03.668503 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-09 19:10:03.668507 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-09 19:10:03.668512 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-09 19:10:03.668516 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-09 19:10:03.668520 | controller | period = 900\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668524 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668528 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account 2026-03-09 19:10:03.668533 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-09 19:10:03.668537 | controller | \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.668541 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-09 19:10:03.668557 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.668562 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.668566 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.668571 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-09 19:10:03.668575 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-09 19:10:03.668579 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-09 19:10:03.668583 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-09 19:10:03.668588 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-09 19:10:03.668592 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.668596 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-09 19:10:03.668600 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-09 19:10:03.668605 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-09 19:10:03.668611 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-09 19:10:03.668615 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-09 19:10:03.668619 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-09 19:10:03.668624 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-09 19:10:03.668629 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-09 19:10:03.668634 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.668640 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-09 19:10:03.668644 | controller | 19:07:08 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-09 19:10:03.668648 | controller | 19:07:08 | watcher/4-deploy-with-precreated-account | test step completed 4-deploy-with-precreated-account\n 2026-03-09 19:10:03.668652 | controller | \ logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher\n 2026-03-09 19:10:03.668657 | controller | \ logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | running command: [sh -c 2026-03-09 19:10:03.668661 | controller | set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service 2026-03-09 19:10:03.668665 | controller | list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.668669 | controller | 19:07:08 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.668674 | controller | -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:07:08 | 2026-03-09 19:10:03.668678 | controller | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:07:11 | 2026-03-09 19:10:03.668682 | controller | watcher/5-cleanup-watcher | + '[' 1 == 0 ']'\n logger.go:42: 19:07:14 | watcher/5-cleanup-watcher 2026-03-09 19:10:03.668686 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.668691 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.668695 | controller | ]\n ]\n logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | + oc exec 2026-03-09 19:10:03.668699 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.668703 | controller | -c Type\n logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n 2026-03-09 19:10:03.668708 | controller | \ logger.go:42: 19:07:16 | watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 2026-03-09 19:10:03.668712 | controller | 19:07:16 | watcher/5-cleanup-watcher | test step completed 5-cleanup-watcher\n logger.go:42: 2026-03-09 19:10:03.668716 | controller | 19:07:16 | watcher | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-cinder\n 2026-03-09 19:10:03.668720 | controller | \ logger.go:42: 19:07:16 | watcher-cinder | Skipping creation of user-supplied 2026-03-09 19:10:03.668725 | controller | namespace: watcher-kuttl-default\n logger.go:42: 19:07:16 | watcher-cinder/0-cleanup-watcher 2026-03-09 19:10:03.668729 | controller | | starting test step 0-cleanup-watcher\n logger.go:42: 19:07:16 | watcher-cinder/0-cleanup-watcher 2026-03-09 19:10:03.668733 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668737 | controller | | starting test step 1-deploy-watcher-no-cinder\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668749 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:07:16 2026-03-09 19:10:03.668753 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo 2026-03-09 19:10:03.668757 | controller | pipefail\n # check that the decision engine correctly detects that there 2026-03-09 19:10:03.668762 | controller | is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.668766 | controller | | grep -c 'Block storage service is not enabled, skipping storage collector')\" 2026-03-09 19:10:03.668770 | controller | == 2 ]\n ]\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668774 | controller | | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.668779 | controller | 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage 2026-03-09 19:10:03.668785 | controller | service is not enabled, skipping storage collector'\n logger.go:42: 19:07:16 2026-03-09 19:10:03.668789 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): 2026-03-09 19:10:03.668794 | controller | pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n 2026-03-09 19:10:03.668798 | controller | \ logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 2026-03-09 19:10:03.668802 | controller | == 2 ']'\n logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668806 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.668810 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.668815 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.668819 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.668823 | controller | 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.668827 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668832 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.668836 | controller | \ logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.668840 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.668844 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668848 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668853 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.668857 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.668861 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.668865 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.668869 | controller | 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.668874 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668878 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.668884 | controller | \ logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.668888 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.668892 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668897 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668901 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.668905 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.668909 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.668913 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.668918 | controller | 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.668922 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668926 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.668933 | controller | \ logger.go:42: 19:07:20 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.668938 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.668942 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:20 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668953 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668958 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.668962 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.668967 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.668971 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.668975 | controller | 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.668981 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.668985 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.668989 | controller | \ logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.668994 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.668998 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669002 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669007 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669013 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669019 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669026 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669032 | controller | 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669037 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669042 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669046 | controller | \ logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.669050 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.669054 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669058 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669063 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669067 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669071 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669075 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669079 | controller | 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669084 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669088 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669094 | controller | \ logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.669098 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.669103 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669107 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669111 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669117 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669122 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669126 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669130 | controller | 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669134 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669139 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669143 | controller | \ logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.669161 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.669174 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669178 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669183 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669187 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669191 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669195 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669200 | controller | 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669204 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669208 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669212 | controller | \ logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.669217 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.669221 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669225 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669229 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669233 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669238 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669242 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669246 | controller | 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669250 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669255 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669259 | controller | \ logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.669266 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-09 19:10:03.669271 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669275 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669281 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669286 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669290 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669294 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669298 | controller | 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669303 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669307 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669311 | controller | \ logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | Error from 2026-03-09 19:10:03.669316 | controller | server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-09 19:10:03.669320 | controller | is waiting to start: ContainerCreating\n logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669324 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669328 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669333 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669337 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669341 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669345 | controller | 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669349 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669354 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669359 | controller | \ logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | Error from 2026-03-09 19:10:03.669364 | controller | server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-09 19:10:03.669375 | controller | is waiting to start: ContainerCreating\n logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669379 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669384 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669388 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669392 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669396 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669401 | controller | 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage 2026-03-09 19:10:03.669405 | controller | service is not enabled, skipping storage collector'\n logger.go:42: 19:07:30 2026-03-09 19:10:03.669409 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669413 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669419 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669424 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669428 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669432 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669436 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669441 | controller | 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669446 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669451 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669455 | controller | \ logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 2026-03-09 19:10:03.669459 | controller | == 2 ']'\n logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669464 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669468 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669472 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669476 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669480 | controller | 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669485 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669489 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669493 | controller | \ logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-09 19:10:03.669497 | controller | == 2 ']'\n logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669501 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669506 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669510 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669514 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669518 | controller | 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669523 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669527 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669531 | controller | \ logger.go:42: 19:07:34 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-09 19:10:03.669535 | controller | == 2 ']'\n logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669539 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669544 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669549 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669554 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669558 | controller | 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669562 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669568 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669579 | controller | \ logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-09 19:10:03.669584 | controller | == 2 ']'\n logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669588 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669593 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669597 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669601 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669605 | controller | 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669610 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669614 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669618 | controller | \ logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-09 19:10:03.669622 | controller | == 2 ']'\n logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669626 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669631 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669635 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669639 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669643 | controller | 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669647 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669652 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669656 | controller | \ logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-09 19:10:03.669660 | controller | == 2 ']'\n logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669664 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669669 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669673 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669677 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669681 | controller | 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669686 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669690 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669694 | controller | \ logger.go:42: 19:07:39 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-09 19:10:03.669699 | controller | == 2 ']'\n logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669703 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669707 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.669711 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-09 19:10:03.669716 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.669720 | controller | 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669726 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669730 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.669736 | controller | \ logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-09 19:10:03.669740 | controller | == 2 ']'\n logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-09 19:10:03.669744 | controller | | test step completed 1-deploy-watcher-no-cinder\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669749 | controller | | starting test step 2-deploy-cinder\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669753 | controller | | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n logger.go:42: 2026-03-09 19:10:03.669757 | controller | 19:07:40 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.669761 | controller | \ # check that the decision detects that there is a cinder service and\n # 2026-03-09 19:10:03.669766 | controller | does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE 2026-03-09 19:10:03.669777 | controller | watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, 2026-03-09 19:10:03.669781 | controller | skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669786 | controller | | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.669790 | controller | 19:07:40 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.669794 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669799 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669803 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669807 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.669811 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.669816 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.669820 | controller | 0 ]\n ]\n logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.669824 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.669828 | controller | 19:07:41 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.669832 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669837 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669841 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669845 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.669849 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.669853 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.669858 | controller | 0 ]\n ]\n logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.669862 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-09 19:10:03.669866 | controller | 19:07:42 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.669870 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669877 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669881 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669885 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.669889 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.669894 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.669898 | controller | 0 ]\n ]\n logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.669902 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.669906 | controller | 19:07:43 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.669911 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:44 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669915 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669919 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669925 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.669932 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.669936 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.669940 | controller | 0 ]\n ]\n logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.669945 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.669949 | controller | 19:07:45 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.669953 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669957 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.669962 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.669966 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.669970 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.669981 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.669985 | controller | 0 ]\n ]\n logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.669990 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.669994 | controller | 19:07:46 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.669998 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670002 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670007 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670013 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670022 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670028 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670034 | controller | 0 ]\n ]\n logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670040 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670048 | controller | 19:07:47 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670054 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670059 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670065 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670070 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670076 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670082 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670088 | controller | 0 ]\n ]\n logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670093 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670098 | controller | 19:07:48 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670102 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:49 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670106 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670110 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670115 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670119 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670123 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670127 | controller | 0 ]\n ]\n logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670134 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670138 | controller | 19:07:50 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670142 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670162 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670166 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670171 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670175 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670179 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670183 | controller | 0 ]\n ]\n logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670187 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670192 | controller | 19:07:51 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670196 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670200 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670204 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670208 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670213 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670217 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670230 | controller | 0 ]\n ]\n logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670236 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670241 | controller | 19:07:52 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670245 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670249 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670254 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670258 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670262 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670266 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670271 | controller | 0 ]\n ]\n logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670275 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670279 | controller | 19:07:53 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670283 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670287 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670292 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670296 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670300 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670304 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670309 | controller | 0 ]\n ]\n logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670313 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670317 | controller | 19:07:54 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670321 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:55 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670326 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670332 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670336 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670341 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670345 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670349 | controller | 0 ]\n ]\n logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670353 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670358 | controller | 19:07:56 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670362 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670366 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670370 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670375 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670379 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670385 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670389 | controller | 0 ]\n ]\n logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670393 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670398 | controller | 19:07:57 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670402 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670406 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670410 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670414 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670419 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670423 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670434 | controller | 0 ]\n ]\n logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670438 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670443 | controller | 19:07:58 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670447 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670451 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670456 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670460 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670464 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670468 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670472 | controller | 0 ]\n ]\n logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670477 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670481 | controller | 19:07:59 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670485 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670489 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670494 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670498 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670502 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670506 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670512 | controller | 0 ]\n ]\n logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670516 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670521 | controller | 19:08:00 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670525 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:01 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670529 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670533 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670539 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670544 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670548 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670552 | controller | 0 ]\n ]\n logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670556 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670561 | controller | 19:08:02 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670565 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670569 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670573 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670578 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670582 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670586 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670590 | controller | 0 ]\n ]\n logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670595 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670599 | controller | 19:08:03 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670603 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670607 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670612 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670616 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670622 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670626 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670637 | controller | 0 ]\n ]\n logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670642 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670646 | controller | 19:08:04 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670651 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670655 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670659 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670663 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670667 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670672 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670676 | controller | 0 ]\n ]\n logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670680 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670684 | controller | 19:08:05 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670689 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670696 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670701 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670705 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670709 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670713 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670718 | controller | 0 ]\n ]\n logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670722 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670726 | controller | 19:08:06 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670730 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:07 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670734 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670739 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670743 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670747 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670751 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670756 | controller | 0 ]\n ]\n logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670760 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-09 19:10:03.670764 | controller | 19:08:08 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.670768 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670772 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670777 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670781 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670785 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670789 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670794 | controller | 0 ]\n ]\n logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670798 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670802 | controller | 19:08:09 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670806 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670810 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670815 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670819 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670823 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670827 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670832 | controller | 0 ]\n ]\n logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670843 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670848 | controller | 19:08:10 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670854 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670858 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670862 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670867 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670871 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670875 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670881 | controller | 0 ]\n ]\n logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670886 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670890 | controller | 19:08:11 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670894 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670898 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670903 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670907 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670911 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670915 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670919 | controller | 0 ]\n ]\n logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670924 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670928 | controller | 19:08:12 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670932 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:13 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670936 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670941 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670945 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670949 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670953 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670957 | controller | 0 ]\n ]\n logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.670962 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.670966 | controller | 19:08:14 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.670970 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670974 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.670979 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.670983 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.670987 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.670991 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.670996 | controller | 0 ]\n ]\n logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671002 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671007 | controller | 19:08:15 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671013 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671019 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671025 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671031 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671037 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671042 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671046 | controller | 0 ]\n ]\n logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671058 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671063 | controller | 19:08:16 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671067 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671075 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671080 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671084 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671088 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671092 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671096 | controller | 0 ]\n ]\n logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671101 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671105 | controller | 19:08:17 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671109 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:18 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671114 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671118 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671122 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671126 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671130 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671135 | controller | 0 ]\n ]\n logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671139 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671143 | controller | 19:08:19 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671161 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671166 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671170 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671174 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671178 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671185 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671189 | controller | 0 ]\n ]\n logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671193 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671197 | controller | 19:08:20 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671205 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671209 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671214 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671218 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671222 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671226 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671231 | controller | 0 ]\n ]\n logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671235 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671239 | controller | 19:08:21 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671243 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671248 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671252 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671256 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671260 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671266 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671271 | controller | 0 ]\n ]\n logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671282 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671287 | controller | 19:08:22 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671291 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671296 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671300 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671304 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671308 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671313 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671317 | controller | 0 ]\n ]\n logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671321 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671325 | controller | 19:08:23 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671330 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:24 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671334 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671338 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671342 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671348 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671353 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671357 | controller | 0 ]\n ]\n logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671361 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671365 | controller | 19:08:25 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671370 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671374 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671378 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671382 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671387 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671393 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671399 | controller | 0 ]\n ]\n logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671405 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671411 | controller | 19:08:26 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671416 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671421 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671425 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671429 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671433 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671438 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671442 | controller | 0 ]\n ]\n logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671446 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671452 | controller | 19:08:27 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671457 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671461 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671465 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671469 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671474 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671478 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671482 | controller | 0 ]\n ]\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671494 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671498 | controller | 19:08:28 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671503 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671507 | controller | | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-09 19:10:03.671513 | controller | is waiting to start: ContainerCreating\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671517 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671522 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671526 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671530 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671534 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671569 | controller | 0 ]\n ]\n logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671574 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671578 | controller | 19:08:29 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671582 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:30 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671587 | controller | | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-09 19:10:03.671591 | controller | is waiting to start: ContainerCreating\n logger.go:42: 19:08:30 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671595 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671599 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671604 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671608 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671612 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671616 | controller | 0 ]\n ]\n logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671621 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671625 | controller | 19:08:31 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671629 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671633 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671638 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671642 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671646 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671650 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671654 | controller | 0 ]\n ]\n logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671659 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-09 19:10:03.671663 | controller | 19:08:32 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.671667 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671671 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671676 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671680 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671684 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671693 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671698 | controller | 0 ]\n ]\n logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671702 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671706 | controller | 19:08:33 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671710 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671715 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671719 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671730 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671734 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671738 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671742 | controller | 0 ]\n ]\n logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671747 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671771 | controller | 19:08:34 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671776 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671781 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671785 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671789 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671795 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671799 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671804 | controller | 0 ]\n ]\n logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671808 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671812 | controller | 19:08:35 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671816 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:36 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671821 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671825 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671829 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671833 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671838 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671842 | controller | 0 ]\n ]\n logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671846 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671850 | controller | 19:08:37 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671855 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671859 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671863 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671869 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671873 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671878 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671882 | controller | 0 ]\n ]\n logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671886 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671892 | controller | 19:08:38 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671896 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671900 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671905 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671909 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671913 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671917 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671921 | controller | 0 ]\n ]\n logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671926 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671930 | controller | 19:08:39 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671934 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671938 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671942 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.671954 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-09 19:10:03.671959 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.671963 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.671968 | controller | 0 ]\n ]\n logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | ++ 2026-03-09 19:10:03.671972 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.671976 | controller | 19:08:40 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.671980 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:41 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671985 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:08:41 | watcher-cinder/2-deploy-cinder 2026-03-09 19:10:03.671989 | controller | | test step completed 2-deploy-cinder\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.671993 | controller | | starting test step 3-remove-cinder\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.671997 | controller | | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n logger.go:42: 2026-03-09 19:10:03.672002 | controller | 19:08:41 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.672007 | controller | \ # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.672013 | controller | service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep 2026-03-09 19:10:03.672019 | controller | -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n 2026-03-09 19:10:03.672025 | controller | \ ]\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | ++ oc logs 2026-03-09 19:10:03.672034 | controller | -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:41 2026-03-09 19:10:03.672038 | controller | | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, 2026-03-09 19:10:03.672043 | controller | skipping storage collector'\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672047 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672051 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672055 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672059 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672064 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672068 | controller | 19:08:42 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672072 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672079 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.672083 | controller | \ logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n 2026-03-09 19:10:03.672087 | controller | \ logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-09 19:10:03.672091 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-09 19:10:03.672096 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.672100 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.672104 | controller | 2 ]\n ]\n logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | ++ 2026-03-09 19:10:03.672108 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-09 19:10:03.672113 | controller | 19:08:43 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672117 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672121 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672125 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672130 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672134 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672138 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672142 | controller | 19:08:44 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.672171 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672175 | controller | | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.672180 | controller | 19:08:45 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n logger.go:42: 2026-03-09 19:10:03.672192 | controller | 19:08:46 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n 2026-03-09 19:10:03.672197 | controller | \ # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.672201 | controller | service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep 2026-03-09 19:10:03.672205 | controller | -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n 2026-03-09 19:10:03.672210 | controller | \ ]\n logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | ++ oc logs 2026-03-09 19:10:03.672217 | controller | -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:46 2026-03-09 19:10:03.672221 | controller | | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, 2026-03-09 19:10:03.672226 | controller | skipping storage collector'\n logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672230 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672234 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672238 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672243 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672247 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672251 | controller | 19:08:48 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672255 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672260 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.672264 | controller | \ logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | Error from server 2026-03-09 19:10:03.672268 | controller | (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-09 19:10:03.672272 | controller | is waiting to start: ContainerCreating\n logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672280 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672284 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672288 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672293 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672297 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672301 | controller | 19:08:49 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672305 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672310 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.672314 | controller | \ logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | Error from server 2026-03-09 19:10:03.672318 | controller | (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-09 19:10:03.672322 | controller | is waiting to start: ContainerCreating\n logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672326 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672331 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672335 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672339 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672343 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672348 | controller | 19:08:50 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672352 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672356 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.672360 | controller | \ logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n 2026-03-09 19:10:03.672364 | controller | \ logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-09 19:10:03.672370 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-09 19:10:03.672375 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.672379 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.672383 | controller | 2 ]\n ]\n logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | ++ 2026-03-09 19:10:03.672388 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.672399 | controller | 19:08:51 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.672403 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672407 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672412 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672416 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672422 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672426 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672430 | controller | 19:08:52 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672435 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672439 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.672443 | controller | \ logger.go:42: 19:08:53 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n 2026-03-09 19:10:03.672447 | controller | \ logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-09 19:10:03.672453 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-09 19:10:03.672458 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.672462 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.672466 | controller | 2 ]\n ]\n logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | ++ 2026-03-09 19:10:03.672470 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.672475 | controller | 19:08:54 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.672480 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672485 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672489 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672493 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672497 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672502 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672506 | controller | 19:08:55 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672510 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672514 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.672519 | controller | \ logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n 2026-03-09 19:10:03.672523 | controller | \ logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-09 19:10:03.672529 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-09 19:10:03.672533 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.672537 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.672542 | controller | 2 ]\n ]\n logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | ++ 2026-03-09 19:10:03.672546 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.672550 | controller | 19:08:56 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.672554 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672559 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672563 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672567 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672571 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672576 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672580 | controller | 19:08:57 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672584 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672588 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.672592 | controller | \ logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n 2026-03-09 19:10:03.672597 | controller | \ logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-09 19:10:03.672607 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-09 19:10:03.672612 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.672616 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-09 19:10:03.672620 | controller | 2 ]\n ]\n logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | ++ 2026-03-09 19:10:03.672626 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-09 19:10:03.672631 | controller | 19:08:58 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-09 19:10:03.672635 | controller | not enabled, skipping storage collector'\n logger.go:42: 19:08:59 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672639 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672644 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-09 19:10:03.672648 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-09 19:10:03.672652 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-09 19:10:03.672656 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-09 19:10:03.672660 | controller | 19:09:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.672665 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder 2026-03-09 19:10:03.672669 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-09 19:10:03.672673 | controller | \ logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n 2026-03-09 19:10:03.672677 | controller | \ logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | test step completed 2026-03-09 19:10:03.672683 | controller | 3-remove-cinder\n logger.go:42: 19:09:00 | watcher-cinder/4-cleanup-watcher | 2026-03-09 19:10:03.672687 | controller | starting test step 4-cleanup-watcher\n logger.go:42: 19:09:08 | watcher-cinder/4-cleanup-watcher 2026-03-09 19:10:03.672692 | controller | | test step completed 4-cleanup-watcher\n logger.go:42: 19:09:08 | watcher-cinder 2026-03-09 19:10:03.672696 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-api-scaling\n 2026-03-09 19:10:03.672700 | controller | \ logger.go:42: 19:09:08 | watcher-api-scaling | Skipping creation of user-supplied 2026-03-09 19:10:03.672704 | controller | namespace: watcher-kuttl-default\n logger.go:42: 19:09:08 | watcher-api-scaling/0-cleanup-watcher 2026-03-09 19:10:03.672709 | controller | | starting test step 0-cleanup-watcher\n logger.go:42: 19:09:08 | watcher-api-scaling/0-cleanup-watcher 2026-03-09 19:10:03.672713 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672717 | controller | | starting test step 1-deploy-with-defaults\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672721 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:09:08 2026-03-09 19:10:03.672726 | controller | | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo 2026-03-09 19:10:03.672730 | controller | pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.672734 | controller | service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc 2026-03-09 19:10:03.672738 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.672742 | controller | -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default 2026-03-09 19:10:03.672747 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-09 19:10:03.672751 | controller | -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-09 19:10:03.672755 | controller | ]\n # If we are running the container locally, skip following test\n if 2026-03-09 19:10:03.672759 | controller | [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-09 19:10:03.672764 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-09 19:10:03.672768 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-09 19:10:03.672772 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-09 19:10:03.672776 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-09 19:10:03.672781 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-09 19:10:03.672785 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-09 19:10:03.672789 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-09 19:10:03.672800 | controller | variables found.\"\n fi\n ]\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672804 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.672808 | controller | value -c Name -c Type\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672813 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:09:11 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672817 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672821 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.672826 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-09 19:10:03.672832 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.672836 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-09 19:10:03.672841 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.672845 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.672851 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-09 19:10:03.672855 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.672859 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.672864 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.672868 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.672872 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.672876 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.672880 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.672885 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.672889 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.672893 | controller | \ ]\n logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672897 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.672902 | controller | value -c Name -c Type\n logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672906 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:09:14 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672910 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672914 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.672918 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-09 19:10:03.672923 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.672927 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-09 19:10:03.672931 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.672935 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.672939 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-09 19:10:03.672944 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.672948 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.672952 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.672956 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.672961 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.672965 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.672969 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.672975 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.672979 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.672984 | controller | \ ]\n logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.672994 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.673000 | controller | value -c Name -c Type\n logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673004 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673010 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673016 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.673022 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673028 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673034 | controller | | ++ grep watcher\n logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673039 | controller | | + SERVICEID=63195c8d1c084032a109516d802e4c7d\n logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673046 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.673051 | controller | \ logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-09 19:10:03.673058 | controller | 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d ']'\n logger.go:42: 2026-03-09 19:10:03.673063 | controller | 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.673069 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:09:20 2026-03-09 19:10:03.673078 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.673084 | controller | ']'\n logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ 2026-03-09 19:10:03.673089 | controller | oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-09 19:10:03.673095 | controller | \ logger.go:42: 19:09:21 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-09 19:10:03.673100 | controller | '' == '' ']'\n logger.go:42: 19:09:21 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673108 | controller | | + exit 0\n logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673114 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.673120 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-09 19:10:03.673126 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.673131 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-09 19:10:03.673135 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.673139 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.673143 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-09 19:10:03.673163 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.673168 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.673174 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.673180 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.673185 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.673191 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.673197 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.673203 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.673208 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.673214 | controller | \ ]\n logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673220 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.673226 | controller | value -c Name -c Type\n logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673232 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673238 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673244 | controller | | ++ grep watcher\n logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673259 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673266 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.673272 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673277 | controller | | + SERVICEID=63195c8d1c084032a109516d802e4c7d\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673283 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.673289 | controller | \ logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-09 19:10:03.673295 | controller | 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d ']'\n logger.go:42: 2026-03-09 19:10:03.673301 | controller | 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.673307 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:09:26 2026-03-09 19:10:03.673313 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.673319 | controller | ']'\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ 2026-03-09 19:10:03.673325 | controller | oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-09 19:10:03.673329 | controller | \ logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-09 19:10:03.673334 | controller | '' == '' ']'\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673338 | controller | | + exit 0\n logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673342 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.673346 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-09 19:10:03.673351 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.673355 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-09 19:10:03.673359 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.673368 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.673372 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-09 19:10:03.673376 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.673380 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.673385 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.673391 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.673396 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.673400 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.673404 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.673409 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.673413 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.673417 | controller | \ ]\n logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673421 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.673425 | controller | value -c Name -c Type\n logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673430 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673434 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673438 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.673442 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673448 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673452 | controller | | ++ grep watcher\n logger.go:42: 19:09:31 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673457 | controller | | + SERVICEID=63195c8d1c084032a109516d802e4c7d\n logger.go:42: 19:09:31 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673461 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.673474 | controller | \ logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-09 19:10:03.673478 | controller | 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d ']'\n logger.go:42: 2026-03-09 19:10:03.673483 | controller | 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.673487 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:09:32 2026-03-09 19:10:03.673491 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.673495 | controller | ']'\n logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | ++ 2026-03-09 19:10:03.673500 | controller | oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-09 19:10:03.673504 | controller | \ logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-09 19:10:03.673508 | controller | '' == '' ']'\n logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673512 | controller | | + exit 0\n logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673519 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.673523 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-09 19:10:03.673528 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.673532 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-09 19:10:03.673536 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.673540 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-09 19:10:03.673544 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-09 19:10:03.673549 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-09 19:10:03.673553 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-09 19:10:03.673557 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-09 19:10:03.673561 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-09 19:10:03.673566 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-09 19:10:03.673570 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-09 19:10:03.673574 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-09 19:10:03.673578 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-09 19:10:03.673583 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-09 19:10:03.673587 | controller | \ ]\n logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673591 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.673595 | controller | value -c Name -c Type\n logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673600 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673604 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673608 | controller | | ++ grep watcher\n logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673612 | controller | | ++ awk '{print $1}'\n logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673616 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.673621 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673625 | controller | | + SERVICEID=63195c8d1c084032a109516d802e4c7d\n logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673629 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-09 19:10:03.673633 | controller | \ logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-09 19:10:03.673638 | controller | 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d ']'\n logger.go:42: 2026-03-09 19:10:03.673642 | controller | 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.673646 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:09:38 2026-03-09 19:10:03.673658 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.673665 | controller | ']'\n logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | ++ 2026-03-09 19:10:03.673669 | controller | oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-09 19:10:03.673673 | controller | \ logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-09 19:10:03.673678 | controller | '' == '' ']'\n logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673682 | controller | | + exit 0\n logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults 2026-03-09 19:10:03.673686 | controller | | test step completed 1-deploy-with-defaults\n logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api 2026-03-09 19:10:03.673691 | controller | | starting test step 2-scale-up-watcher-api\n logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api 2026-03-09 19:10:03.673695 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-09 19:10:03.673699 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":3}]'\n 2026-03-09 19:10:03.673703 | controller | \ ]\n logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api 2026-03-09 19:10:03.673707 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:09:49 2026-03-09 19:10:03.673712 | controller | | watcher-api-scaling/2-scale-up-watcher-api | test step completed 2-scale-up-watcher-api\n 2026-03-09 19:10:03.673716 | controller | \ logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | starting 2026-03-09 19:10:03.673720 | controller | test step 3-scale-down-watcher-api\n logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api 2026-03-09 19:10:03.673724 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-09 19:10:03.673729 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":1}]'\n 2026-03-09 19:10:03.673733 | controller | \ ]\n logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api 2026-03-09 19:10:03.673737 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:09:52 2026-03-09 19:10:03.673741 | controller | | watcher-api-scaling/3-scale-down-watcher-api | test step completed 3-scale-down-watcher-api\n 2026-03-09 19:10:03.673746 | controller | \ logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api | 2026-03-09 19:10:03.673750 | controller | starting test step 4-scale-down-zero-watcher-api\n logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-09 19:10:03.673754 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-09 19:10:03.673758 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":0}]'\n 2026-03-09 19:10:03.673762 | controller | \ ]\n logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-09 19:10:03.673767 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:09:53 2026-03-09 19:10:03.673771 | controller | | watcher-api-scaling/4-scale-down-zero-watcher-api | test step completed 4-scale-down-zero-watcher-api\n 2026-03-09 19:10:03.673775 | controller | \ logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | starting test 2026-03-09 19:10:03.673779 | controller | step 5-cleanup-watcher\n logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673784 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.673788 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.673792 | controller | ]\n ]\n logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673796 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.673800 | controller | value -c Name -c Type\n logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673806 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:09:56 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673811 | controller | | + '[' 1 == 0 ']'\n logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673815 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.673819 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.673823 | controller | ]\n ]\n logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673828 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.673834 | controller | value -c Name -c Type\n logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673838 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:09:59 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673842 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673853 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-09 19:10:03.673857 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-09 19:10:03.673862 | controller | ]\n ]\n logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673866 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-09 19:10:03.673870 | controller | value -c Name -c Type\n logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673874 | controller | | ++ grep -c '^watcher'\n logger.go:42: 19:10:02 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673879 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 19:10:02 | watcher-api-scaling/5-cleanup-watcher 2026-03-09 19:10:03.673883 | controller | | test step completed 5-cleanup-watcher\n logger.go:42: 19:10:02 | watcher-api-scaling 2026-03-09 19:10:03.673887 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/deps\n logger.go:42: 2026-03-09 19:10:03.673892 | controller | 19:10:02 | deps | Ignoring infra.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-09 19:10:03.673896 | controller | \ logger.go:42: 19:10:02 | deps | Ignoring keystone.yaml as it does not match 2026-03-09 19:10:03.673900 | controller | file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 19:10:02 | 2026-03-09 19:10:03.673904 | controller | deps | Ignoring kustomization.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-09 19:10:03.673910 | controller | \ logger.go:42: 19:10:02 | deps | Ignoring namespace.yaml as it does not match 2026-03-09 19:10:03.673916 | controller | file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 19:10:02 | 2026-03-09 19:10:03.673920 | controller | deps | Ignoring telemetry.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-09 19:10:03.673925 | controller | \ logger.go:42: 19:10:02 | deps | Skipping creation of user-supplied namespace: 2026-03-09 19:10:03.673929 | controller | watcher-kuttl-default\n logger.go:42: 19:10:02 | deps | skipping kubernetes event 2026-03-09 19:10:03.673933 | controller | logging\n=== CONT kuttl/harness/common\n logger.go:42: 19:10:02 | common | Ignoring 2026-03-09 19:10:03.673937 | controller | cleanup-assert.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-09 19:10:03.673942 | controller | \ logger.go:42: 19:10:02 | common | Ignoring cleanup-errors.yaml as it does not 2026-03-09 19:10:03.673946 | controller | match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 19:10:02 2026-03-09 19:10:03.673950 | controller | | common | Ignoring cleanup-watcher.yaml as it does not match file name regexp: 2026-03-09 19:10:03.673955 | controller | ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 19:10:02 | common | Ignoring 2026-03-09 19:10:03.673960 | controller | deploy-with-defaults.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-09 19:10:03.673965 | controller | \ logger.go:42: 19:10:02 | common | Skipping creation of user-supplied namespace: 2026-03-09 19:10:03.673969 | controller | watcher-kuttl-default\n logger.go:42: 19:10:02 | common | skipping kubernetes 2026-03-09 19:10:03.673973 | controller | event logging\n=== NAME kuttl\n harness.go:406: run tests finished\n harness.go:514: 2026-03-09 19:10:03.673978 | controller | cleaning up\n harness.go:571: removing temp folder: \"\"\n--- FAIL: kuttl (1502.65s)\n 2026-03-09 19:10:03.673982 | controller | \ --- FAIL: kuttl/harness (0.00s)\n --- FAIL: kuttl/harness/appcred-tests 2026-03-09 19:10:03.673986 | controller | (959.40s)\n --- PASS: kuttl/harness/watcher-notification (36.28s)\n --- 2026-03-09 19:10:03.673990 | controller | PASS: kuttl/harness/watcher-topology (33.59s)\n --- PASS: kuttl/harness/watcher-tls-certs-change 2026-03-09 19:10:03.673995 | controller | (37.89s)\n --- PASS: kuttl/harness/watcher-tls (120.05s)\n --- PASS: 2026-03-09 19:10:03.673999 | controller | kuttl/harness/watcher-rmquser (34.79s)\n --- PASS: kuttl/harness/watcher 2026-03-09 19:10:03.674003 | controller | (114.14s)\n --- PASS: kuttl/harness/watcher-cinder (112.06s)\n --- 2026-03-09 19:10:03.674009 | controller | PASS: kuttl/harness/watcher-api-scaling (54.41s)\n --- PASS: kuttl/harness/deps 2026-03-09 19:10:03.674015 | controller | (0.00s)\n --- PASS: kuttl/harness/common (0.00s)\nFAIL" 2026-03-09 19:10:03.674021 | controller | stdout_lines: 2026-03-09 19:10:03.674028 | controller | - oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml 2026-03-09 19:10:03.674034 | controller | - === RUN kuttl 2026-03-09 19:10:03.674039 | controller | - ' harness.go:463: starting setup' 2026-03-09 19:10:03.674044 | controller | - ' harness.go:255: running tests using configured kubeconfig.' 2026-03-09 19:10:03.674048 | controller | - ' harness.go:278: Successful connection to cluster at: https://api.crc.testing:6443' 2026-03-09 19:10:03.674052 | controller | - ' harness.go:363: running tests' 2026-03-09 19:10:03.674057 | controller | - ' harness.go:75: going to run test suite with timeout of 300 seconds for each 2026-03-09 19:10:03.674061 | controller | step' 2026-03-09 19:10:03.674066 | controller | - ' harness.go:375: testsuite: test/kuttl/test-suites/default/ has 11 tests' 2026-03-09 19:10:03.674070 | controller | - === RUN kuttl/harness 2026-03-09 19:10:03.674074 | controller | - === RUN kuttl/harness/appcred-tests 2026-03-09 19:10:03.674079 | controller | - === PAUSE kuttl/harness/appcred-tests 2026-03-09 19:10:03.674090 | controller | - === RUN kuttl/harness/common 2026-03-09 19:10:03.674095 | controller | - === PAUSE kuttl/harness/common 2026-03-09 19:10:03.674099 | controller | - === RUN kuttl/harness/deps 2026-03-09 19:10:03.674103 | controller | - === PAUSE kuttl/harness/deps 2026-03-09 19:10:03.674108 | controller | - === RUN kuttl/harness/watcher 2026-03-09 19:10:03.674112 | controller | - === PAUSE kuttl/harness/watcher 2026-03-09 19:10:03.674116 | controller | - === RUN kuttl/harness/watcher-api-scaling 2026-03-09 19:10:03.674120 | controller | - === PAUSE kuttl/harness/watcher-api-scaling 2026-03-09 19:10:03.674125 | controller | - === RUN kuttl/harness/watcher-cinder 2026-03-09 19:10:03.674129 | controller | - === PAUSE kuttl/harness/watcher-cinder 2026-03-09 19:10:03.674133 | controller | - === RUN kuttl/harness/watcher-notification 2026-03-09 19:10:03.674137 | controller | - === PAUSE kuttl/harness/watcher-notification 2026-03-09 19:10:03.674142 | controller | - === RUN kuttl/harness/watcher-rmquser 2026-03-09 19:10:03.674161 | controller | - === PAUSE kuttl/harness/watcher-rmquser 2026-03-09 19:10:03.674165 | controller | - === RUN kuttl/harness/watcher-tls 2026-03-09 19:10:03.674169 | controller | - === PAUSE kuttl/harness/watcher-tls 2026-03-09 19:10:03.674174 | controller | - === RUN kuttl/harness/watcher-tls-certs-change 2026-03-09 19:10:03.674178 | controller | - === PAUSE kuttl/harness/watcher-tls-certs-change 2026-03-09 19:10:03.674182 | controller | - === RUN kuttl/harness/watcher-topology 2026-03-09 19:10:03.674190 | controller | - === PAUSE kuttl/harness/watcher-topology 2026-03-09 19:10:03.674195 | controller | - === CONT kuttl/harness/appcred-tests 2026-03-09 19:10:03.674199 | controller | - ' logger.go:42: 18:45:00 | appcred-tests | Skipping creation of user-supplied 2026-03-09 19:10:03.674205 | controller | namespace: watcher-kuttl-default' 2026-03-09 19:10:03.674210 | controller | - ' logger.go:42: 18:45:00 | appcred-tests/0-cleanup | starting test step 0-cleanup' 2026-03-09 19:10:03.674214 | controller | - ' logger.go:42: 18:45:00 | appcred-tests/0-cleanup | test step completed 0-cleanup' 2026-03-09 19:10:03.674218 | controller | - ' logger.go:42: 18:45:00 | appcred-tests/1-deploy | starting test step 1-deploy' 2026-03-09 19:10:03.674223 | controller | - ' logger.go:42: 18:45:00 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.674227 | controller | created' 2026-03-09 19:10:03.674234 | controller | - ' logger.go:42: 18:45:00 | appcred-tests/1-deploy | running command: [sh -c set 2026-03-09 19:10:03.674238 | controller | -euo pipefail' 2026-03-09 19:10:03.674243 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.674247 | controller | - ' oc wait -n "${NS}" watcher watcher-kuttl --for=condition=Ready --timeout=600s]' 2026-03-09 19:10:03.674251 | controller | - ' logger.go:42: 18:45:57 | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.674256 | controller | condition met' 2026-03-09 19:10:03.674260 | controller | - ' logger.go:42: 18:45:57 | appcred-tests/1-deploy | test step completed 1-deploy' 2026-03-09 19:10:03.674264 | controller | - ' logger.go:42: 18:45:57 | appcred-tests/2-deploy-appcred | starting test step 2026-03-09 19:10:03.674268 | controller | 2-deploy-appcred' 2026-03-09 19:10:03.674273 | controller | - ' logger.go:42: 18:45:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.674277 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.674281 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.674285 | controller | - ' ' 2026-03-09 19:10:03.674290 | controller | - ' old_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674294 | controller | - ' old_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674298 | controller | - ' old_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674302 | controller | - ' oc create configmap appcred-watcher-pre \' 2026-03-09 19:10:03.674307 | controller | - ' --from-literal=api_uid="${old_api_uid}" \' 2026-03-09 19:10:03.674311 | controller | - ' --from-literal=applier_uid="${old_applier_uid}" \' 2026-03-09 19:10:03.674315 | controller | - ' --from-literal=decision_engine_uid="${old_de_uid}" \' 2026-03-09 19:10:03.674320 | controller | - ' --dry-run=client -o yaml | oc apply -n "${NS}" -f -' 2026-03-09 19:10:03.674324 | controller | - ' ' 2026-03-09 19:10:03.674330 | controller | - ' cat <<''EOF'' | oc apply -n "${NS}" -f -' 2026-03-09 19:10:03.674334 | controller | - ' apiVersion: keystone.openstack.org/v1beta1' 2026-03-09 19:10:03.674338 | controller | - ' kind: KeystoneApplicationCredential' 2026-03-09 19:10:03.674343 | controller | - ' metadata:' 2026-03-09 19:10:03.674347 | controller | - ' name: ac-watcher' 2026-03-09 19:10:03.674351 | controller | - ' spec:' 2026-03-09 19:10:03.674355 | controller | - ' secret: osp-secret' 2026-03-09 19:10:03.674360 | controller | - ' passwordSelector: WatcherPassword' 2026-03-09 19:10:03.674364 | controller | - ' userName: watcher' 2026-03-09 19:10:03.674368 | controller | - ' roles:' 2026-03-09 19:10:03.674373 | controller | - ' - admin' 2026-03-09 19:10:03.674377 | controller | - ' - service' 2026-03-09 19:10:03.674381 | controller | - ' unrestricted: false' 2026-03-09 19:10:03.674385 | controller | - ' EOF' 2026-03-09 19:10:03.674390 | controller | - ' ' 2026-03-09 19:10:03.674394 | controller | - ' oc patch watcher watcher-kuttl -n "${NS}" --type=merge -p ''{"spec":{"auth":{"applicationCredentialSecret":"ac-watcher-secret"}}}'']' 2026-03-09 19:10:03.674398 | controller | - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre 2026-03-09 19:10:03.674406 | controller | created' 2026-03-09 19:10:03.674412 | controller | - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher 2026-03-09 19:10:03.674418 | controller | created' 2026-03-09 19:10:03.674424 | controller | - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.674430 | controller | patched' 2026-03-09 19:10:03.674435 | controller | - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.674441 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.674447 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.674453 | controller | - ' ' 2026-03-09 19:10:03.674468 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.674475 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.674480 | controller | - ' ' 2026-03-09 19:10:03.674484 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.674489 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.674493 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.674497 | controller | - ' exit 1' 2026-03-09 19:10:03.674501 | controller | - ' fi' 2026-03-09 19:10:03.674506 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.674510 | controller | - ' ' 2026-03-09 19:10:03.674514 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.674519 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.674523 | controller | | base64 -d)' 2026-03-09 19:10:03.674527 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.674531 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.674536 | controller | - ' exit 1' 2026-03-09 19:10:03.674540 | controller | - ' fi' 2026-03-09 19:10:03.674544 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.674548 | controller | - ' ' 2026-03-09 19:10:03.674553 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.674557 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.674561 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.674566 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.674570 | controller | --timeout=300s' 2026-03-09 19:10:03.674574 | controller | - ' ' 2026-03-09 19:10:03.674578 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.674583 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.674587 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.674591 | controller | - ' ' 2026-03-09 19:10:03.674596 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674600 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674604 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674608 | controller | - ' ' 2026-03-09 19:10:03.674613 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.674617 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.674621 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.674629 | controller | available"' 2026-03-09 19:10:03.674633 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.674638 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.674642 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.674646 | controller | - ' exit 1' 2026-03-09 19:10:03.674650 | controller | - ' fi' 2026-03-09 19:10:03.674655 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.674659 | controller | - ' ' 2026-03-09 19:10:03.674663 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.674667 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.674672 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.674676 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.674680 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.674684 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.674689 | controller | -- \' 2026-03-09 19:10:03.674693 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.674697 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.674702 | controller | - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.674706 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.674710 | controller | - ' logger.go:42: 18:50:58 | appcred-tests/2-deploy-appcred | error: timed out 2026-03-09 19:10:03.674714 | controller | waiting for the condition on keystoneapplicationcredentials/ac-watcher' 2026-03-09 19:10:03.674719 | controller | - ' logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.674723 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.674727 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.674732 | controller | - ' ' 2026-03-09 19:10:03.674736 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.674748 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.674752 | controller | - ' ' 2026-03-09 19:10:03.674757 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.674761 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.674765 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.674770 | controller | - ' exit 1' 2026-03-09 19:10:03.674774 | controller | - ' fi' 2026-03-09 19:10:03.674778 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.674782 | controller | - ' ' 2026-03-09 19:10:03.674787 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.674791 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.674795 | controller | | base64 -d)' 2026-03-09 19:10:03.674800 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.674804 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.674808 | controller | - ' exit 1' 2026-03-09 19:10:03.674812 | controller | - ' fi' 2026-03-09 19:10:03.674817 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.674821 | controller | - ' ' 2026-03-09 19:10:03.674825 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.674831 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.674836 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.674840 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.674844 | controller | --timeout=300s' 2026-03-09 19:10:03.674848 | controller | - ' ' 2026-03-09 19:10:03.674853 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.674857 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.674861 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.674865 | controller | - ' ' 2026-03-09 19:10:03.674870 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674876 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674880 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.674884 | controller | - ' ' 2026-03-09 19:10:03.674889 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.674893 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.674897 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.674902 | controller | available"' 2026-03-09 19:10:03.674906 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.674910 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.674914 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.674919 | controller | - ' exit 1' 2026-03-09 19:10:03.674923 | controller | - ' fi' 2026-03-09 19:10:03.674927 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.674932 | controller | - ' ' 2026-03-09 19:10:03.674936 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.674940 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.674944 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.674949 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.674953 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.674957 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.674962 | controller | -- \' 2026-03-09 19:10:03.674966 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.674970 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.674976 | controller | - ' logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.674981 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.674985 | controller | - ' logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.674989 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.674994 | controller | found' 2026-03-09 19:10:03.674998 | controller | - ' logger.go:42: 18:51:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.675002 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.675010 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.675016 | controller | - ' ' 2026-03-09 19:10:03.675022 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.675037 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.675042 | controller | - ' ' 2026-03-09 19:10:03.675046 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.675051 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.675055 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.675060 | controller | - ' exit 1' 2026-03-09 19:10:03.675064 | controller | - ' fi' 2026-03-09 19:10:03.675068 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.675075 | controller | - ' ' 2026-03-09 19:10:03.675079 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.675083 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.675088 | controller | | base64 -d)' 2026-03-09 19:10:03.675092 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.675096 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.675101 | controller | - ' exit 1' 2026-03-09 19:10:03.675107 | controller | - ' fi' 2026-03-09 19:10:03.675111 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.675115 | controller | - ' ' 2026-03-09 19:10:03.675120 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.675124 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.675128 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.675133 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.675137 | controller | --timeout=300s' 2026-03-09 19:10:03.675141 | controller | - ' ' 2026-03-09 19:10:03.675162 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.675169 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.675174 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.675178 | controller | - ' ' 2026-03-09 19:10:03.675182 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675186 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675191 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675195 | controller | - ' ' 2026-03-09 19:10:03.675199 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.675204 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.675208 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.675212 | controller | available"' 2026-03-09 19:10:03.675216 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.675221 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.675225 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.675229 | controller | - ' exit 1' 2026-03-09 19:10:03.675233 | controller | - ' fi' 2026-03-09 19:10:03.675238 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.675244 | controller | - ' ' 2026-03-09 19:10:03.675249 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.675253 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.675257 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675261 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.675266 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675270 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.675274 | controller | -- \' 2026-03-09 19:10:03.675279 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675283 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.675287 | controller | - ' logger.go:42: 18:51:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.675292 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.675296 | controller | - ' logger.go:42: 18:51:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.675300 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.675305 | controller | found' 2026-03-09 19:10:03.675309 | controller | - ' logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.675313 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.675317 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.675322 | controller | - ' ' 2026-03-09 19:10:03.675326 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.675330 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.675335 | controller | - ' ' 2026-03-09 19:10:03.675347 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.675351 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.675356 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.675360 | controller | - ' exit 1' 2026-03-09 19:10:03.675364 | controller | - ' fi' 2026-03-09 19:10:03.675370 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.675374 | controller | - ' ' 2026-03-09 19:10:03.675381 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.675386 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.675390 | controller | | base64 -d)' 2026-03-09 19:10:03.675394 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.675399 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.675403 | controller | - ' exit 1' 2026-03-09 19:10:03.675407 | controller | - ' fi' 2026-03-09 19:10:03.675411 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.675416 | controller | - ' ' 2026-03-09 19:10:03.675420 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.675424 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.675429 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.675433 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.675437 | controller | --timeout=300s' 2026-03-09 19:10:03.675441 | controller | - ' ' 2026-03-09 19:10:03.675446 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.675454 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.675460 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.675464 | controller | - ' ' 2026-03-09 19:10:03.675469 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675473 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675477 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675482 | controller | - ' ' 2026-03-09 19:10:03.675486 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.675490 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.675495 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.675499 | controller | available"' 2026-03-09 19:10:03.675503 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.675507 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.675512 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.675516 | controller | - ' exit 1' 2026-03-09 19:10:03.675520 | controller | - ' fi' 2026-03-09 19:10:03.675524 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.675529 | controller | - ' ' 2026-03-09 19:10:03.675533 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.675537 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.675542 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675546 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.675550 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675554 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.675559 | controller | -- \' 2026-03-09 19:10:03.675563 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675567 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.675571 | controller | - ' logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.675576 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.675580 | controller | - ' logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.675584 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.675588 | controller | found' 2026-03-09 19:10:03.675593 | controller | - ' logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.675597 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.675601 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.675606 | controller | - ' ' 2026-03-09 19:10:03.675610 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.675614 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.675619 | controller | - ' ' 2026-03-09 19:10:03.675630 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.675634 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.675640 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.675644 | controller | - ' exit 1' 2026-03-09 19:10:03.675649 | controller | - ' fi' 2026-03-09 19:10:03.675653 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.675657 | controller | - ' ' 2026-03-09 19:10:03.675662 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.675666 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.675670 | controller | | base64 -d)' 2026-03-09 19:10:03.675675 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.675679 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.675683 | controller | - ' exit 1' 2026-03-09 19:10:03.675688 | controller | - ' fi' 2026-03-09 19:10:03.675692 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.675696 | controller | - ' ' 2026-03-09 19:10:03.675701 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.675705 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.675711 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.675715 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.675719 | controller | --timeout=300s' 2026-03-09 19:10:03.675724 | controller | - ' ' 2026-03-09 19:10:03.675728 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.675732 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.675736 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.675741 | controller | - ' ' 2026-03-09 19:10:03.675745 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675768 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675774 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.675778 | controller | - ' ' 2026-03-09 19:10:03.675782 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.675787 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.675791 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.675795 | controller | available"' 2026-03-09 19:10:03.675800 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.675804 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.675808 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.675813 | controller | - ' exit 1' 2026-03-09 19:10:03.675817 | controller | - ' fi' 2026-03-09 19:10:03.675821 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.675825 | controller | - ' ' 2026-03-09 19:10:03.675830 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.675834 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.675838 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675842 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.675847 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675853 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.675857 | controller | -- \' 2026-03-09 19:10:03.675862 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.675866 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.675870 | controller | - ' logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.675875 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.675879 | controller | - ' logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.675883 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.675887 | controller | found' 2026-03-09 19:10:03.675892 | controller | - ' logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.675896 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.675900 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.675905 | controller | - ' ' 2026-03-09 19:10:03.675909 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.675913 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.675918 | controller | - ' ' 2026-03-09 19:10:03.675922 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.675934 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.675938 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.675942 | controller | - ' exit 1' 2026-03-09 19:10:03.675947 | controller | - ' fi' 2026-03-09 19:10:03.675951 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.675956 | controller | - ' ' 2026-03-09 19:10:03.675960 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.675964 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.675968 | controller | | base64 -d)' 2026-03-09 19:10:03.675973 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.675977 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.675981 | controller | - ' exit 1' 2026-03-09 19:10:03.675985 | controller | - ' fi' 2026-03-09 19:10:03.675990 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.675994 | controller | - ' ' 2026-03-09 19:10:03.675998 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.676003 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.676008 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.676014 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.676020 | controller | --timeout=300s' 2026-03-09 19:10:03.676026 | controller | - ' ' 2026-03-09 19:10:03.676032 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.676038 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.676042 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.676047 | controller | - ' ' 2026-03-09 19:10:03.676051 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676055 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676062 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676066 | controller | - ' ' 2026-03-09 19:10:03.676073 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.676078 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.676082 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.676086 | controller | available"' 2026-03-09 19:10:03.676090 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.676095 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.676099 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.676103 | controller | - ' exit 1' 2026-03-09 19:10:03.676108 | controller | - ' fi' 2026-03-09 19:10:03.676112 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.676116 | controller | - ' ' 2026-03-09 19:10:03.676120 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.676125 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.676129 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676133 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.676137 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676142 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.676168 | controller | -- \' 2026-03-09 19:10:03.676172 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676177 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.676181 | controller | - ' logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.676185 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.676190 | controller | - ' logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.676194 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.676198 | controller | found' 2026-03-09 19:10:03.676202 | controller | - ' logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.676207 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.676211 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.676215 | controller | - ' ' 2026-03-09 19:10:03.676220 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.676224 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.676228 | controller | - ' ' 2026-03-09 19:10:03.676233 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.676237 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.676249 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.676254 | controller | - ' exit 1' 2026-03-09 19:10:03.676258 | controller | - ' fi' 2026-03-09 19:10:03.676263 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.676267 | controller | - ' ' 2026-03-09 19:10:03.676271 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.676276 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.676283 | controller | | base64 -d)' 2026-03-09 19:10:03.676287 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.676292 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.676296 | controller | - ' exit 1' 2026-03-09 19:10:03.676300 | controller | - ' fi' 2026-03-09 19:10:03.676305 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.676309 | controller | - ' ' 2026-03-09 19:10:03.676313 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.676317 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.676322 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.676329 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.676333 | controller | --timeout=300s' 2026-03-09 19:10:03.676337 | controller | - ' ' 2026-03-09 19:10:03.676342 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.676346 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.676350 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.676354 | controller | - ' ' 2026-03-09 19:10:03.676360 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676364 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676369 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676373 | controller | - ' ' 2026-03-09 19:10:03.676377 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.676382 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.676386 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.676390 | controller | available"' 2026-03-09 19:10:03.676395 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.676399 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.676403 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.676407 | controller | - ' exit 1' 2026-03-09 19:10:03.676411 | controller | - ' fi' 2026-03-09 19:10:03.676416 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.676420 | controller | - ' ' 2026-03-09 19:10:03.676424 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.676428 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.676433 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676437 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.676443 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676447 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.676451 | controller | -- \' 2026-03-09 19:10:03.676456 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676460 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.676464 | controller | - ' logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.676470 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.676475 | controller | - ' logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.676479 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.676483 | controller | found' 2026-03-09 19:10:03.676487 | controller | - ' logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.676492 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.676496 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.676500 | controller | - ' ' 2026-03-09 19:10:03.676505 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.676509 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.676513 | controller | - ' ' 2026-03-09 19:10:03.676517 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.676522 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.676526 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.676537 | controller | - ' exit 1' 2026-03-09 19:10:03.676542 | controller | - ' fi' 2026-03-09 19:10:03.676546 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.676550 | controller | - ' ' 2026-03-09 19:10:03.676554 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.676559 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.676563 | controller | | base64 -d)' 2026-03-09 19:10:03.676567 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.676571 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.676576 | controller | - ' exit 1' 2026-03-09 19:10:03.676580 | controller | - ' fi' 2026-03-09 19:10:03.676584 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.676588 | controller | - ' ' 2026-03-09 19:10:03.676593 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.676597 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.676601 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.676605 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.676610 | controller | --timeout=300s' 2026-03-09 19:10:03.676614 | controller | - ' ' 2026-03-09 19:10:03.676618 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.676622 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.676627 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.676631 | controller | - ' ' 2026-03-09 19:10:03.676635 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676639 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676644 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676648 | controller | - ' ' 2026-03-09 19:10:03.676652 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.676656 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.676661 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.676665 | controller | available"' 2026-03-09 19:10:03.676671 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.676677 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.676681 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.676686 | controller | - ' exit 1' 2026-03-09 19:10:03.676690 | controller | - ' fi' 2026-03-09 19:10:03.676694 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.676699 | controller | - ' ' 2026-03-09 19:10:03.676703 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.676707 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.676711 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676716 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.676720 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676724 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.676728 | controller | -- \' 2026-03-09 19:10:03.676733 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676737 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.676741 | controller | - ' logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.676745 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.676750 | controller | - ' logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.676754 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.676758 | controller | found' 2026-03-09 19:10:03.676762 | controller | - ' logger.go:42: 18:51:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.676767 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.676771 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.676775 | controller | - ' ' 2026-03-09 19:10:03.676779 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.676784 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.676788 | controller | - ' ' 2026-03-09 19:10:03.676792 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.676796 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.676801 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.676805 | controller | - ' exit 1' 2026-03-09 19:10:03.676809 | controller | - ' fi' 2026-03-09 19:10:03.676820 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.676825 | controller | - ' ' 2026-03-09 19:10:03.676829 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.676834 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.676838 | controller | | base64 -d)' 2026-03-09 19:10:03.676842 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.676847 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.676851 | controller | - ' exit 1' 2026-03-09 19:10:03.676855 | controller | - ' fi' 2026-03-09 19:10:03.676860 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.676864 | controller | - ' ' 2026-03-09 19:10:03.676868 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.676872 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.676879 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.676883 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.676887 | controller | --timeout=300s' 2026-03-09 19:10:03.676892 | controller | - ' ' 2026-03-09 19:10:03.676896 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.676900 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.676905 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.676909 | controller | - ' ' 2026-03-09 19:10:03.676913 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676917 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676922 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.676926 | controller | - ' ' 2026-03-09 19:10:03.676932 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.676936 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.676940 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.676945 | controller | available"' 2026-03-09 19:10:03.676949 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.676953 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.676957 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.676962 | controller | - ' exit 1' 2026-03-09 19:10:03.676966 | controller | - ' fi' 2026-03-09 19:10:03.676970 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.676974 | controller | - ' ' 2026-03-09 19:10:03.676979 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.676983 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.676987 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.676991 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.676996 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677000 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.677005 | controller | -- \' 2026-03-09 19:10:03.677011 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677017 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.677023 | controller | - ' logger.go:42: 18:51:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.677029 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.677035 | controller | - ' logger.go:42: 18:51:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.677040 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.677044 | controller | found' 2026-03-09 19:10:03.677048 | controller | - ' logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.677053 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.677057 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.677063 | controller | - ' ' 2026-03-09 19:10:03.677068 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.677072 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.677076 | controller | - ' ' 2026-03-09 19:10:03.677080 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.677085 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.677089 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.677093 | controller | - ' exit 1' 2026-03-09 19:10:03.677098 | controller | - ' fi' 2026-03-09 19:10:03.677102 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.677114 | controller | - ' ' 2026-03-09 19:10:03.677119 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.677123 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.677128 | controller | | base64 -d)' 2026-03-09 19:10:03.677132 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.677136 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.677141 | controller | - ' exit 1' 2026-03-09 19:10:03.677157 | controller | - ' fi' 2026-03-09 19:10:03.677168 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.677172 | controller | - ' ' 2026-03-09 19:10:03.677176 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.677181 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.677185 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.677189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.677193 | controller | --timeout=300s' 2026-03-09 19:10:03.677198 | controller | - ' ' 2026-03-09 19:10:03.677202 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.677206 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.677211 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.677215 | controller | - ' ' 2026-03-09 19:10:03.677219 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677223 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677228 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677232 | controller | - ' ' 2026-03-09 19:10:03.677236 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.677240 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.677245 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.677249 | controller | available"' 2026-03-09 19:10:03.677253 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.677258 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.677262 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.677266 | controller | - ' exit 1' 2026-03-09 19:10:03.677270 | controller | - ' fi' 2026-03-09 19:10:03.677275 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.677279 | controller | - ' ' 2026-03-09 19:10:03.677285 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.677289 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.677294 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677298 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.677302 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677306 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.677311 | controller | -- \' 2026-03-09 19:10:03.677315 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677319 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.677323 | controller | - ' logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.677328 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.677334 | controller | - ' logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.677350 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.677355 | controller | found' 2026-03-09 19:10:03.677359 | controller | - ' logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.677363 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.677368 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.677372 | controller | - ' ' 2026-03-09 19:10:03.677377 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.677381 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.677385 | controller | - ' ' 2026-03-09 19:10:03.677390 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.677394 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.677398 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.677402 | controller | - ' exit 1' 2026-03-09 19:10:03.677407 | controller | - ' fi' 2026-03-09 19:10:03.677411 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.677415 | controller | - ' ' 2026-03-09 19:10:03.677427 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.677432 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.677436 | controller | | base64 -d)' 2026-03-09 19:10:03.677440 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.677445 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.677449 | controller | - ' exit 1' 2026-03-09 19:10:03.677453 | controller | - ' fi' 2026-03-09 19:10:03.677458 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.677462 | controller | - ' ' 2026-03-09 19:10:03.677466 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.677471 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.677475 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.677479 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.677483 | controller | --timeout=300s' 2026-03-09 19:10:03.677488 | controller | - ' ' 2026-03-09 19:10:03.677492 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.677497 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.677504 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.677509 | controller | - ' ' 2026-03-09 19:10:03.677513 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677519 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677523 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677528 | controller | - ' ' 2026-03-09 19:10:03.677532 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.677536 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.677542 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.677546 | controller | available"' 2026-03-09 19:10:03.677551 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.677555 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.677559 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.677563 | controller | - ' exit 1' 2026-03-09 19:10:03.677567 | controller | - ' fi' 2026-03-09 19:10:03.677572 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.677576 | controller | - ' ' 2026-03-09 19:10:03.677580 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.677584 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.677589 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677593 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.677597 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677602 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.677606 | controller | -- \' 2026-03-09 19:10:03.677610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677614 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.677619 | controller | - ' logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.677623 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.677627 | controller | - ' logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.677631 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.677636 | controller | found' 2026-03-09 19:10:03.677640 | controller | - ' logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.677644 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.677648 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.677653 | controller | - ' ' 2026-03-09 19:10:03.677657 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.677661 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.677665 | controller | - ' ' 2026-03-09 19:10:03.677670 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.677674 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.677678 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.677684 | controller | - ' exit 1' 2026-03-09 19:10:03.677688 | controller | - ' fi' 2026-03-09 19:10:03.677693 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.677697 | controller | - ' ' 2026-03-09 19:10:03.677701 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.677713 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.677718 | controller | | base64 -d)' 2026-03-09 19:10:03.677722 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.677728 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.677732 | controller | - ' exit 1' 2026-03-09 19:10:03.677738 | controller | - ' fi' 2026-03-09 19:10:03.677742 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.677746 | controller | - ' ' 2026-03-09 19:10:03.677751 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.677755 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.677759 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.677763 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.677768 | controller | --timeout=300s' 2026-03-09 19:10:03.677772 | controller | - ' ' 2026-03-09 19:10:03.677776 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.677781 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.677785 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.677789 | controller | - ' ' 2026-03-09 19:10:03.677794 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677798 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677802 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.677806 | controller | - ' ' 2026-03-09 19:10:03.677810 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.677815 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.677819 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.677823 | controller | available"' 2026-03-09 19:10:03.677827 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.677832 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.677836 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.677840 | controller | - ' exit 1' 2026-03-09 19:10:03.677844 | controller | - ' fi' 2026-03-09 19:10:03.677849 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.677853 | controller | - ' ' 2026-03-09 19:10:03.677857 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.677862 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.677866 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677870 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.677874 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677880 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.677884 | controller | -- \' 2026-03-09 19:10:03.677889 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.677893 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.677897 | controller | - ' logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.677901 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.677906 | controller | - ' logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.677910 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.677914 | controller | found' 2026-03-09 19:10:03.677918 | controller | - ' logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.677923 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.677927 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.677931 | controller | - ' ' 2026-03-09 19:10:03.677936 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.677940 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.677944 | controller | - ' ' 2026-03-09 19:10:03.677948 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.677953 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.677957 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.677961 | controller | - ' exit 1' 2026-03-09 19:10:03.677965 | controller | - ' fi' 2026-03-09 19:10:03.677970 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.677974 | controller | - ' ' 2026-03-09 19:10:03.677980 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.677991 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.677996 | controller | | base64 -d)' 2026-03-09 19:10:03.678000 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.678005 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.678011 | controller | - ' exit 1' 2026-03-09 19:10:03.678017 | controller | - ' fi' 2026-03-09 19:10:03.678023 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.678030 | controller | - ' ' 2026-03-09 19:10:03.678036 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.678040 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.678044 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.678049 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.678053 | controller | --timeout=300s' 2026-03-09 19:10:03.678057 | controller | - ' ' 2026-03-09 19:10:03.678061 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.678066 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.678070 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.678074 | controller | - ' ' 2026-03-09 19:10:03.678078 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678082 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678087 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678093 | controller | - ' ' 2026-03-09 19:10:03.678097 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.678102 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.678106 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.678110 | controller | available"' 2026-03-09 19:10:03.678115 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.678121 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.678125 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.678130 | controller | - ' exit 1' 2026-03-09 19:10:03.678134 | controller | - ' fi' 2026-03-09 19:10:03.678138 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.678143 | controller | - ' ' 2026-03-09 19:10:03.678161 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.678166 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.678170 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678174 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.678178 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678183 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.678187 | controller | -- \' 2026-03-09 19:10:03.678191 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678195 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.678200 | controller | - ' logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.678204 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.678208 | controller | - ' logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.678212 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.678217 | controller | found' 2026-03-09 19:10:03.678221 | controller | - ' logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.678225 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.678229 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.678234 | controller | - ' ' 2026-03-09 19:10:03.678238 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.678242 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.678246 | controller | - ' ' 2026-03-09 19:10:03.678251 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.678255 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.678259 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.678264 | controller | - ' exit 1' 2026-03-09 19:10:03.678274 | controller | - ' fi' 2026-03-09 19:10:03.678279 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.678283 | controller | - ' ' 2026-03-09 19:10:03.678287 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.678291 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.678296 | controller | | base64 -d)' 2026-03-09 19:10:03.678308 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.678315 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.678319 | controller | - ' exit 1' 2026-03-09 19:10:03.678324 | controller | - ' fi' 2026-03-09 19:10:03.678328 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.678332 | controller | - ' ' 2026-03-09 19:10:03.678337 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.678341 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.678345 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.678350 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.678354 | controller | --timeout=300s' 2026-03-09 19:10:03.678358 | controller | - ' ' 2026-03-09 19:10:03.678363 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.678367 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.678371 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.678375 | controller | - ' ' 2026-03-09 19:10:03.678380 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678384 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678388 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678392 | controller | - ' ' 2026-03-09 19:10:03.678397 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.678401 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.678405 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.678410 | controller | available"' 2026-03-09 19:10:03.678414 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.678418 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.678422 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.678427 | controller | - ' exit 1' 2026-03-09 19:10:03.678431 | controller | - ' fi' 2026-03-09 19:10:03.678435 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.678440 | controller | - ' ' 2026-03-09 19:10:03.678444 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.678448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.678452 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678457 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.678461 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678465 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.678471 | controller | -- \' 2026-03-09 19:10:03.678476 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678480 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.678484 | controller | - ' logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.678490 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.678495 | controller | - ' logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.678518 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.678523 | controller | found' 2026-03-09 19:10:03.678527 | controller | - ' logger.go:42: 18:51:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.678531 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.678535 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.678540 | controller | - ' ' 2026-03-09 19:10:03.678544 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.678548 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.678553 | controller | - ' ' 2026-03-09 19:10:03.678557 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.678561 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.678565 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.678570 | controller | - ' exit 1' 2026-03-09 19:10:03.678574 | controller | - ' fi' 2026-03-09 19:10:03.678578 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.678583 | controller | - ' ' 2026-03-09 19:10:03.678587 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.678591 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.678595 | controller | | base64 -d)' 2026-03-09 19:10:03.678600 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.678611 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.678615 | controller | - ' exit 1' 2026-03-09 19:10:03.678620 | controller | - ' fi' 2026-03-09 19:10:03.678624 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.678628 | controller | - ' ' 2026-03-09 19:10:03.678632 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.678637 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.678641 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.678645 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.678650 | controller | --timeout=300s' 2026-03-09 19:10:03.678654 | controller | - ' ' 2026-03-09 19:10:03.678658 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.678662 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.678667 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.678671 | controller | - ' ' 2026-03-09 19:10:03.678675 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678680 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678684 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678688 | controller | - ' ' 2026-03-09 19:10:03.678692 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.678697 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.678701 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.678705 | controller | available"' 2026-03-09 19:10:03.678709 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.678716 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.678720 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.678724 | controller | - ' exit 1' 2026-03-09 19:10:03.678729 | controller | - ' fi' 2026-03-09 19:10:03.678733 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.678737 | controller | - ' ' 2026-03-09 19:10:03.678744 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.678748 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.678752 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678756 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.678761 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678765 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.678769 | controller | -- \' 2026-03-09 19:10:03.678773 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.678778 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.678782 | controller | - ' logger.go:42: 18:51:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.678786 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.678791 | controller | - ' logger.go:42: 18:51:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.678795 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.678799 | controller | found' 2026-03-09 19:10:03.678803 | controller | - ' logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.678808 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.678812 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.678816 | controller | - ' ' 2026-03-09 19:10:03.678820 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.678825 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.678829 | controller | - ' ' 2026-03-09 19:10:03.678833 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.678837 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.678842 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.678846 | controller | - ' exit 1' 2026-03-09 19:10:03.678850 | controller | - ' fi' 2026-03-09 19:10:03.678854 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.678859 | controller | - ' ' 2026-03-09 19:10:03.678863 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.678867 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.678871 | controller | | base64 -d)' 2026-03-09 19:10:03.678876 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.678887 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.678891 | controller | - ' exit 1' 2026-03-09 19:10:03.678895 | controller | - ' fi' 2026-03-09 19:10:03.678900 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.678904 | controller | - ' ' 2026-03-09 19:10:03.678908 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.678912 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.678921 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.678925 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.678929 | controller | --timeout=300s' 2026-03-09 19:10:03.678934 | controller | - ' ' 2026-03-09 19:10:03.678938 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.678942 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.678946 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.678951 | controller | - ' ' 2026-03-09 19:10:03.678955 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678959 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678963 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.678968 | controller | - ' ' 2026-03-09 19:10:03.678972 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.678976 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.678980 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.678985 | controller | available"' 2026-03-09 19:10:03.678989 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.678993 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.678997 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.679002 | controller | - ' exit 1' 2026-03-09 19:10:03.679007 | controller | - ' fi' 2026-03-09 19:10:03.679013 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.679019 | controller | - ' ' 2026-03-09 19:10:03.679025 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.679032 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.679037 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679041 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.679045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679050 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.679054 | controller | -- \' 2026-03-09 19:10:03.679058 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679062 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.679069 | controller | - ' logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.679073 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.679078 | controller | - ' logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.679082 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.679086 | controller | found' 2026-03-09 19:10:03.679091 | controller | - ' logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.679095 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.679099 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.679104 | controller | - ' ' 2026-03-09 19:10:03.679110 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.679114 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.679118 | controller | - ' ' 2026-03-09 19:10:03.679123 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.679127 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.679131 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.679135 | controller | - ' exit 1' 2026-03-09 19:10:03.679140 | controller | - ' fi' 2026-03-09 19:10:03.679160 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.679168 | controller | - ' ' 2026-03-09 19:10:03.679172 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.679177 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.679181 | controller | | base64 -d)' 2026-03-09 19:10:03.679185 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.679190 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.679202 | controller | - ' exit 1' 2026-03-09 19:10:03.679207 | controller | - ' fi' 2026-03-09 19:10:03.679211 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.679215 | controller | - ' ' 2026-03-09 19:10:03.679220 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.679224 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.679228 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.679232 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.679237 | controller | --timeout=300s' 2026-03-09 19:10:03.679243 | controller | - ' ' 2026-03-09 19:10:03.679248 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.679252 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.679256 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.679260 | controller | - ' ' 2026-03-09 19:10:03.679265 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679269 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679273 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679277 | controller | - ' ' 2026-03-09 19:10:03.679282 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.679286 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.679290 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.679294 | controller | available"' 2026-03-09 19:10:03.679299 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.679303 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.679307 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.679311 | controller | - ' exit 1' 2026-03-09 19:10:03.679318 | controller | - ' fi' 2026-03-09 19:10:03.679323 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.679327 | controller | - ' ' 2026-03-09 19:10:03.679331 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.679337 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.679342 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679346 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.679350 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679354 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.679359 | controller | -- \' 2026-03-09 19:10:03.679363 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679367 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.679372 | controller | - ' logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.679376 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.679382 | controller | - ' logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.679386 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.679390 | controller | found' 2026-03-09 19:10:03.679394 | controller | - ' logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.679399 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.679403 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.679407 | controller | - ' ' 2026-03-09 19:10:03.679412 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.679416 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.679420 | controller | - ' ' 2026-03-09 19:10:03.679424 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.679429 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.679433 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.679437 | controller | - ' exit 1' 2026-03-09 19:10:03.679441 | controller | - ' fi' 2026-03-09 19:10:03.679446 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.679450 | controller | - ' ' 2026-03-09 19:10:03.679454 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.679458 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.679463 | controller | | base64 -d)' 2026-03-09 19:10:03.679467 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.679471 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.679475 | controller | - ' exit 1' 2026-03-09 19:10:03.679480 | controller | - ' fi' 2026-03-09 19:10:03.679491 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.679495 | controller | - ' ' 2026-03-09 19:10:03.679499 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.679504 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.679508 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.679512 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.679516 | controller | --timeout=300s' 2026-03-09 19:10:03.679521 | controller | - ' ' 2026-03-09 19:10:03.679525 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.679529 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.679535 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.679539 | controller | - ' ' 2026-03-09 19:10:03.679544 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679548 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679554 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679558 | controller | - ' ' 2026-03-09 19:10:03.679562 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.679567 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.679571 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.679575 | controller | available"' 2026-03-09 19:10:03.679579 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.679584 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.679588 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.679592 | controller | - ' exit 1' 2026-03-09 19:10:03.679596 | controller | - ' fi' 2026-03-09 19:10:03.679601 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.679605 | controller | - ' ' 2026-03-09 19:10:03.679609 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.679613 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.679618 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679622 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.679626 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679630 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.679635 | controller | -- \' 2026-03-09 19:10:03.679639 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679643 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.679647 | controller | - ' logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.679652 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.679656 | controller | - ' logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.679660 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.679664 | controller | found' 2026-03-09 19:10:03.679669 | controller | - ' logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.679673 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.679677 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.679681 | controller | - ' ' 2026-03-09 19:10:03.679687 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.679692 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.679697 | controller | - ' ' 2026-03-09 19:10:03.679703 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.679708 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.679714 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.679720 | controller | - ' exit 1' 2026-03-09 19:10:03.679728 | controller | - ' fi' 2026-03-09 19:10:03.679734 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.679740 | controller | - ' ' 2026-03-09 19:10:03.679765 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.679772 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.679778 | controller | | base64 -d)' 2026-03-09 19:10:03.679782 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.679786 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.679791 | controller | - ' exit 1' 2026-03-09 19:10:03.679795 | controller | - ' fi' 2026-03-09 19:10:03.679799 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.679812 | controller | - ' ' 2026-03-09 19:10:03.679817 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.679821 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.679825 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.679829 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.679834 | controller | --timeout=300s' 2026-03-09 19:10:03.679838 | controller | - ' ' 2026-03-09 19:10:03.679842 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.679846 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.679851 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.679855 | controller | - ' ' 2026-03-09 19:10:03.679859 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679863 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679868 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.679872 | controller | - ' ' 2026-03-09 19:10:03.679876 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.679881 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.679885 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.679889 | controller | available"' 2026-03-09 19:10:03.679893 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.679898 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.679902 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.679906 | controller | - ' exit 1' 2026-03-09 19:10:03.679910 | controller | - ' fi' 2026-03-09 19:10:03.679915 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.679919 | controller | - ' ' 2026-03-09 19:10:03.679923 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.679928 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.679932 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679936 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.679940 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679945 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.679951 | controller | -- \' 2026-03-09 19:10:03.679956 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.679963 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.679967 | controller | - ' logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.679971 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.679976 | controller | - ' logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.679980 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.679984 | controller | found' 2026-03-09 19:10:03.679988 | controller | - ' logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.679993 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.679997 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.680001 | controller | - ' ' 2026-03-09 19:10:03.680007 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.680012 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.680018 | controller | - ' ' 2026-03-09 19:10:03.680025 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.680031 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.680036 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.680040 | controller | - ' exit 1' 2026-03-09 19:10:03.680044 | controller | - ' fi' 2026-03-09 19:10:03.680049 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.680053 | controller | - ' ' 2026-03-09 19:10:03.680057 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.680061 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.680066 | controller | | base64 -d)' 2026-03-09 19:10:03.680070 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.680074 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.680078 | controller | - ' exit 1' 2026-03-09 19:10:03.680083 | controller | - ' fi' 2026-03-09 19:10:03.680087 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.680091 | controller | - ' ' 2026-03-09 19:10:03.680103 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.680108 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.680112 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.680117 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.680121 | controller | --timeout=300s' 2026-03-09 19:10:03.680125 | controller | - ' ' 2026-03-09 19:10:03.680129 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.680134 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.680138 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.680142 | controller | - ' ' 2026-03-09 19:10:03.680163 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680167 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680172 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680178 | controller | - ' ' 2026-03-09 19:10:03.680183 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.680187 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.680191 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.680195 | controller | available"' 2026-03-09 19:10:03.680200 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.680204 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.680208 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.680213 | controller | - ' exit 1' 2026-03-09 19:10:03.680217 | controller | - ' fi' 2026-03-09 19:10:03.680221 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.680225 | controller | - ' ' 2026-03-09 19:10:03.680230 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.680234 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.680238 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680242 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.680247 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680251 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.680255 | controller | -- \' 2026-03-09 19:10:03.680260 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680264 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.680268 | controller | - ' logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.680272 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.680292 | controller | - ' logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.680296 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.680300 | controller | found' 2026-03-09 19:10:03.680305 | controller | - ' logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.680309 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.680313 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.680318 | controller | - ' ' 2026-03-09 19:10:03.680322 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.680326 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.680330 | controller | - ' ' 2026-03-09 19:10:03.680335 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.680339 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.680343 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.680347 | controller | - ' exit 1' 2026-03-09 19:10:03.680352 | controller | - ' fi' 2026-03-09 19:10:03.680356 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.680360 | controller | - ' ' 2026-03-09 19:10:03.680365 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.680369 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.680373 | controller | | base64 -d)' 2026-03-09 19:10:03.680377 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.680382 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.680389 | controller | - ' exit 1' 2026-03-09 19:10:03.680394 | controller | - ' fi' 2026-03-09 19:10:03.680398 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.680402 | controller | - ' ' 2026-03-09 19:10:03.680407 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.680418 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.680423 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.680427 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.680431 | controller | --timeout=300s' 2026-03-09 19:10:03.680438 | controller | - ' ' 2026-03-09 19:10:03.680442 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.680447 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.680451 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.680455 | controller | - ' ' 2026-03-09 19:10:03.680462 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680466 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680471 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680475 | controller | - ' ' 2026-03-09 19:10:03.680479 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.680484 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.680488 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.680492 | controller | available"' 2026-03-09 19:10:03.680496 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.680501 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.680505 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.680509 | controller | - ' exit 1' 2026-03-09 19:10:03.680513 | controller | - ' fi' 2026-03-09 19:10:03.680518 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.680522 | controller | - ' ' 2026-03-09 19:10:03.680526 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.680530 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.680534 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680539 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.680543 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680547 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.680551 | controller | -- \' 2026-03-09 19:10:03.680556 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680560 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.680566 | controller | - ' logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.680570 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.680574 | controller | - ' logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.680582 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.680586 | controller | found' 2026-03-09 19:10:03.680591 | controller | - ' logger.go:42: 18:51:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.680595 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.680599 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.680604 | controller | - ' ' 2026-03-09 19:10:03.680608 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.680612 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.680617 | controller | - ' ' 2026-03-09 19:10:03.680621 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.680625 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.680629 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.680634 | controller | - ' exit 1' 2026-03-09 19:10:03.680638 | controller | - ' fi' 2026-03-09 19:10:03.680642 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.680646 | controller | - ' ' 2026-03-09 19:10:03.680651 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.680655 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.680659 | controller | | base64 -d)' 2026-03-09 19:10:03.680664 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.680668 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.680674 | controller | - ' exit 1' 2026-03-09 19:10:03.680678 | controller | - ' fi' 2026-03-09 19:10:03.680682 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.680686 | controller | - ' ' 2026-03-09 19:10:03.680691 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.680695 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.680707 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.680711 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.680715 | controller | --timeout=300s' 2026-03-09 19:10:03.680720 | controller | - ' ' 2026-03-09 19:10:03.680724 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.680729 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.680733 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.680737 | controller | - ' ' 2026-03-09 19:10:03.680741 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680746 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680750 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.680754 | controller | - ' ' 2026-03-09 19:10:03.680758 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.680763 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.680767 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.680771 | controller | available"' 2026-03-09 19:10:03.680775 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.680780 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.680786 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.680790 | controller | - ' exit 1' 2026-03-09 19:10:03.680794 | controller | - ' fi' 2026-03-09 19:10:03.680799 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.680803 | controller | - ' ' 2026-03-09 19:10:03.680807 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.680812 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.680816 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680820 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.680824 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680829 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.680833 | controller | -- \' 2026-03-09 19:10:03.680837 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.680842 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.680846 | controller | - ' logger.go:42: 18:51:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.680850 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.680855 | controller | - ' logger.go:42: 18:51:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.680859 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.680863 | controller | found' 2026-03-09 19:10:03.680867 | controller | - ' logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.680872 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.680876 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.680882 | controller | - ' ' 2026-03-09 19:10:03.680887 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.680891 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.680895 | controller | - ' ' 2026-03-09 19:10:03.680900 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.680904 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.680908 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.680913 | controller | - ' exit 1' 2026-03-09 19:10:03.680917 | controller | - ' fi' 2026-03-09 19:10:03.680922 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.680926 | controller | - ' ' 2026-03-09 19:10:03.680930 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.680934 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.680939 | controller | | base64 -d)' 2026-03-09 19:10:03.680943 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.680947 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.680952 | controller | - ' exit 1' 2026-03-09 19:10:03.680956 | controller | - ' fi' 2026-03-09 19:10:03.680960 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.680965 | controller | - ' ' 2026-03-09 19:10:03.680969 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.680973 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.680985 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.680992 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.680996 | controller | --timeout=300s' 2026-03-09 19:10:03.681001 | controller | - ' ' 2026-03-09 19:10:03.681005 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.681012 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.681018 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.681024 | controller | - ' ' 2026-03-09 19:10:03.681030 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681036 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681041 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681045 | controller | - ' ' 2026-03-09 19:10:03.681050 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.681054 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.681058 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.681063 | controller | available"' 2026-03-09 19:10:03.681067 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.681071 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.681075 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.681080 | controller | - ' exit 1' 2026-03-09 19:10:03.681084 | controller | - ' fi' 2026-03-09 19:10:03.681088 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.681093 | controller | - ' ' 2026-03-09 19:10:03.681097 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.681101 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.681105 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681110 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.681114 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681118 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.681123 | controller | -- \' 2026-03-09 19:10:03.681127 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681131 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.681136 | controller | - ' logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.681140 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.681156 | controller | - ' logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.681165 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.681170 | controller | found' 2026-03-09 19:10:03.681174 | controller | - ' logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.681178 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.681182 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.681187 | controller | - ' ' 2026-03-09 19:10:03.681191 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.681198 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.681203 | controller | - ' ' 2026-03-09 19:10:03.681207 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.681211 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.681216 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.681220 | controller | - ' exit 1' 2026-03-09 19:10:03.681224 | controller | - ' fi' 2026-03-09 19:10:03.681228 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.681233 | controller | - ' ' 2026-03-09 19:10:03.681237 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.681241 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.681246 | controller | | base64 -d)' 2026-03-09 19:10:03.681250 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.681254 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.681258 | controller | - ' exit 1' 2026-03-09 19:10:03.681263 | controller | - ' fi' 2026-03-09 19:10:03.681267 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.681271 | controller | - ' ' 2026-03-09 19:10:03.681275 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.681280 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.681284 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.681297 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.681301 | controller | --timeout=300s' 2026-03-09 19:10:03.681306 | controller | - ' ' 2026-03-09 19:10:03.681310 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.681315 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.681319 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.681323 | controller | - ' ' 2026-03-09 19:10:03.681327 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681332 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681336 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681340 | controller | - ' ' 2026-03-09 19:10:03.681344 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.681349 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.681353 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.681357 | controller | available"' 2026-03-09 19:10:03.681362 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.681366 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.681370 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.681374 | controller | - ' exit 1' 2026-03-09 19:10:03.681379 | controller | - ' fi' 2026-03-09 19:10:03.681383 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.681387 | controller | - ' ' 2026-03-09 19:10:03.681391 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.681396 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.681402 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681406 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.681410 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.681419 | controller | -- \' 2026-03-09 19:10:03.681423 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681427 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.681432 | controller | - ' logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.681436 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.681440 | controller | - ' logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.681445 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.681449 | controller | found' 2026-03-09 19:10:03.681453 | controller | - ' logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.681457 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.681462 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.681466 | controller | - ' ' 2026-03-09 19:10:03.681470 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.681475 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.681479 | controller | - ' ' 2026-03-09 19:10:03.681483 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.681487 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.681492 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.681496 | controller | - ' exit 1' 2026-03-09 19:10:03.681500 | controller | - ' fi' 2026-03-09 19:10:03.681505 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.681509 | controller | - ' ' 2026-03-09 19:10:03.681516 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.681520 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.681524 | controller | | base64 -d)' 2026-03-09 19:10:03.681529 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.681533 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.681537 | controller | - ' exit 1' 2026-03-09 19:10:03.681542 | controller | - ' fi' 2026-03-09 19:10:03.681546 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.681550 | controller | - ' ' 2026-03-09 19:10:03.681554 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.681559 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.681563 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.681574 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.681578 | controller | --timeout=300s' 2026-03-09 19:10:03.681583 | controller | - ' ' 2026-03-09 19:10:03.681587 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.681591 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.681596 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.681602 | controller | - ' ' 2026-03-09 19:10:03.681606 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681611 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681615 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681619 | controller | - ' ' 2026-03-09 19:10:03.681624 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.681632 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.681636 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.681640 | controller | available"' 2026-03-09 19:10:03.681645 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.681649 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.681653 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.681657 | controller | - ' exit 1' 2026-03-09 19:10:03.681662 | controller | - ' fi' 2026-03-09 19:10:03.681666 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.681670 | controller | - ' ' 2026-03-09 19:10:03.681674 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.681679 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.681683 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681687 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.681692 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681696 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.681700 | controller | -- \' 2026-03-09 19:10:03.681704 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681709 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.681713 | controller | - ' logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.681717 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.681722 | controller | - ' logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.681726 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.681730 | controller | found' 2026-03-09 19:10:03.681736 | controller | - ' logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.681740 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.681744 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.681749 | controller | - ' ' 2026-03-09 19:10:03.681753 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.681757 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.681762 | controller | - ' ' 2026-03-09 19:10:03.681766 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.681770 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.681774 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.681779 | controller | - ' exit 1' 2026-03-09 19:10:03.681783 | controller | - ' fi' 2026-03-09 19:10:03.681789 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.681793 | controller | - ' ' 2026-03-09 19:10:03.681798 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.681802 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.681806 | controller | | base64 -d)' 2026-03-09 19:10:03.681810 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.681816 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.681820 | controller | - ' exit 1' 2026-03-09 19:10:03.681825 | controller | - ' fi' 2026-03-09 19:10:03.681829 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.681833 | controller | - ' ' 2026-03-09 19:10:03.681837 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.681842 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.681846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.681850 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.681854 | controller | --timeout=300s' 2026-03-09 19:10:03.681866 | controller | - ' ' 2026-03-09 19:10:03.681871 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.681875 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.681879 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.681884 | controller | - ' ' 2026-03-09 19:10:03.681888 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681892 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681897 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.681901 | controller | - ' ' 2026-03-09 19:10:03.681905 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.681909 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.681914 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.681918 | controller | available"' 2026-03-09 19:10:03.681922 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.681928 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.681933 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.681937 | controller | - ' exit 1' 2026-03-09 19:10:03.681941 | controller | - ' fi' 2026-03-09 19:10:03.681945 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.681950 | controller | - ' ' 2026-03-09 19:10:03.681954 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.681958 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.681963 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681967 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.681971 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681975 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.681981 | controller | -- \' 2026-03-09 19:10:03.681986 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.681990 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.681994 | controller | - ' logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.681999 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.682003 | controller | - ' logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.682009 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.682015 | controller | found' 2026-03-09 19:10:03.682021 | controller | - ' logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.682027 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.682033 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.682038 | controller | - ' ' 2026-03-09 19:10:03.682042 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.682047 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.682051 | controller | - ' ' 2026-03-09 19:10:03.682055 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.682059 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.682064 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.682068 | controller | - ' exit 1' 2026-03-09 19:10:03.682072 | controller | - ' fi' 2026-03-09 19:10:03.682077 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.682081 | controller | - ' ' 2026-03-09 19:10:03.682085 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.682089 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.682094 | controller | | base64 -d)' 2026-03-09 19:10:03.682098 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.682102 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.682106 | controller | - ' exit 1' 2026-03-09 19:10:03.682111 | controller | - ' fi' 2026-03-09 19:10:03.682115 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.682119 | controller | - ' ' 2026-03-09 19:10:03.682123 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.682128 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.682132 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.682136 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.682140 | controller | --timeout=300s' 2026-03-09 19:10:03.682157 | controller | - ' ' 2026-03-09 19:10:03.682171 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.682176 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.682180 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.682185 | controller | - ' ' 2026-03-09 19:10:03.682189 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682193 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682198 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682202 | controller | - ' ' 2026-03-09 19:10:03.682208 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.682213 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.682217 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.682221 | controller | available"' 2026-03-09 19:10:03.682226 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.682230 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.682234 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.682238 | controller | - ' exit 1' 2026-03-09 19:10:03.682242 | controller | - ' fi' 2026-03-09 19:10:03.682247 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.682251 | controller | - ' ' 2026-03-09 19:10:03.682255 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.682259 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.682264 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682268 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.682272 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682276 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.682281 | controller | -- \' 2026-03-09 19:10:03.682285 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682289 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.682293 | controller | - ' logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.682298 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.682302 | controller | - ' logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.682306 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.682310 | controller | found' 2026-03-09 19:10:03.682315 | controller | - ' logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.682319 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.682323 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.682327 | controller | - ' ' 2026-03-09 19:10:03.682334 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.682338 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.682342 | controller | - ' ' 2026-03-09 19:10:03.682346 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.682351 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.682355 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.682359 | controller | - ' exit 1' 2026-03-09 19:10:03.682364 | controller | - ' fi' 2026-03-09 19:10:03.682368 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.682372 | controller | - ' ' 2026-03-09 19:10:03.682376 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.682380 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.682385 | controller | | base64 -d)' 2026-03-09 19:10:03.682389 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.682393 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.682399 | controller | - ' exit 1' 2026-03-09 19:10:03.682403 | controller | - ' fi' 2026-03-09 19:10:03.682408 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.682412 | controller | - ' ' 2026-03-09 19:10:03.682416 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.682421 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.682425 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.682429 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.682433 | controller | --timeout=300s' 2026-03-09 19:10:03.682437 | controller | - ' ' 2026-03-09 19:10:03.682449 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.682454 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.682458 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.682462 | controller | - ' ' 2026-03-09 19:10:03.682467 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682471 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682475 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682479 | controller | - ' ' 2026-03-09 19:10:03.682484 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.682488 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.682492 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.682497 | controller | available"' 2026-03-09 19:10:03.682501 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.682505 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.682509 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.682514 | controller | - ' exit 1' 2026-03-09 19:10:03.682518 | controller | - ' fi' 2026-03-09 19:10:03.682522 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.682527 | controller | - ' ' 2026-03-09 19:10:03.682533 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.682537 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.682541 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682546 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.682550 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682554 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.682559 | controller | -- \' 2026-03-09 19:10:03.682563 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682567 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.682571 | controller | - ' logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.682576 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.682580 | controller | - ' logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.682586 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.682591 | controller | found' 2026-03-09 19:10:03.682595 | controller | - ' logger.go:42: 18:51:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.682599 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.682604 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.682608 | controller | - ' ' 2026-03-09 19:10:03.682612 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.682616 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.682621 | controller | - ' ' 2026-03-09 19:10:03.682625 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.682629 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.682634 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.682638 | controller | - ' exit 1' 2026-03-09 19:10:03.682642 | controller | - ' fi' 2026-03-09 19:10:03.682646 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.682651 | controller | - ' ' 2026-03-09 19:10:03.682655 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.682659 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.682664 | controller | | base64 -d)' 2026-03-09 19:10:03.682668 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.682672 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.682676 | controller | - ' exit 1' 2026-03-09 19:10:03.682681 | controller | - ' fi' 2026-03-09 19:10:03.682685 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.682689 | controller | - ' ' 2026-03-09 19:10:03.682693 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.682698 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.682702 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.682706 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.682710 | controller | --timeout=300s' 2026-03-09 19:10:03.682715 | controller | - ' ' 2026-03-09 19:10:03.682719 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.682730 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.682736 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.682740 | controller | - ' ' 2026-03-09 19:10:03.682745 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682749 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682753 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.682758 | controller | - ' ' 2026-03-09 19:10:03.682762 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.682766 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.682770 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.682775 | controller | available"' 2026-03-09 19:10:03.682779 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.682783 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.682789 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.682793 | controller | - ' exit 1' 2026-03-09 19:10:03.682798 | controller | - ' fi' 2026-03-09 19:10:03.682802 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.682806 | controller | - ' ' 2026-03-09 19:10:03.682811 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.682815 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.682819 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682823 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.682828 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682832 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.682836 | controller | -- \' 2026-03-09 19:10:03.682840 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.682845 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.682849 | controller | - ' logger.go:42: 18:51:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.682853 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.682857 | controller | - ' logger.go:42: 18:51:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.682862 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.682866 | controller | found' 2026-03-09 19:10:03.682870 | controller | - ' logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.682874 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.682879 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.682883 | controller | - ' ' 2026-03-09 19:10:03.682887 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.682893 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.682897 | controller | - ' ' 2026-03-09 19:10:03.682902 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.682906 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.682910 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.682914 | controller | - ' exit 1' 2026-03-09 19:10:03.682919 | controller | - ' fi' 2026-03-09 19:10:03.682923 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.682927 | controller | - ' ' 2026-03-09 19:10:03.682932 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.682936 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.682940 | controller | | base64 -d)' 2026-03-09 19:10:03.682944 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.682949 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.682953 | controller | - ' exit 1' 2026-03-09 19:10:03.682957 | controller | - ' fi' 2026-03-09 19:10:03.682961 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.682966 | controller | - ' ' 2026-03-09 19:10:03.682970 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.682974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.682979 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.682985 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.682989 | controller | --timeout=300s' 2026-03-09 19:10:03.682993 | controller | - ' ' 2026-03-09 19:10:03.682997 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.683010 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.683016 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.683022 | controller | - ' ' 2026-03-09 19:10:03.683029 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683035 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683040 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683044 | controller | - ' ' 2026-03-09 19:10:03.683048 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.683052 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.683057 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.683061 | controller | available"' 2026-03-09 19:10:03.683065 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.683070 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.683074 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.683078 | controller | - ' exit 1' 2026-03-09 19:10:03.683082 | controller | - ' fi' 2026-03-09 19:10:03.683087 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.683091 | controller | - ' ' 2026-03-09 19:10:03.683095 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.683100 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.683104 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683108 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.683112 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683117 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.683121 | controller | -- \' 2026-03-09 19:10:03.683125 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683130 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.683138 | controller | - ' logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.683142 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.683168 | controller | - ' logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.683173 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.683177 | controller | found' 2026-03-09 19:10:03.683181 | controller | - ' logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.683186 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.683190 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.683194 | controller | - ' ' 2026-03-09 19:10:03.683199 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.683203 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.683210 | controller | - ' ' 2026-03-09 19:10:03.683214 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.683218 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.683222 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.683227 | controller | - ' exit 1' 2026-03-09 19:10:03.683231 | controller | - ' fi' 2026-03-09 19:10:03.683235 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.683240 | controller | - ' ' 2026-03-09 19:10:03.683246 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.683250 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.683255 | controller | | base64 -d)' 2026-03-09 19:10:03.683259 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.683263 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.683268 | controller | - ' exit 1' 2026-03-09 19:10:03.683272 | controller | - ' fi' 2026-03-09 19:10:03.683276 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.683280 | controller | - ' ' 2026-03-09 19:10:03.683285 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.683289 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.683293 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.683297 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.683302 | controller | --timeout=300s' 2026-03-09 19:10:03.683308 | controller | - ' ' 2026-03-09 19:10:03.683312 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.683316 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.683329 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.683333 | controller | - ' ' 2026-03-09 19:10:03.683338 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683342 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683346 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683351 | controller | - ' ' 2026-03-09 19:10:03.683355 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.683359 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.683363 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.683368 | controller | available"' 2026-03-09 19:10:03.683372 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.683376 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.683381 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.683385 | controller | - ' exit 1' 2026-03-09 19:10:03.683389 | controller | - ' fi' 2026-03-09 19:10:03.683393 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.683398 | controller | - ' ' 2026-03-09 19:10:03.683402 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.683406 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.683412 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683417 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.683421 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683425 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.683430 | controller | -- \' 2026-03-09 19:10:03.683434 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683438 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.683442 | controller | - ' logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.683447 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.683451 | controller | - ' logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.683455 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.683460 | controller | found' 2026-03-09 19:10:03.683464 | controller | - ' logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.683468 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.683473 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.683477 | controller | - ' ' 2026-03-09 19:10:03.683481 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.683485 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.683490 | controller | - ' ' 2026-03-09 19:10:03.683494 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.683498 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.683502 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.683507 | controller | - ' exit 1' 2026-03-09 19:10:03.683513 | controller | - ' fi' 2026-03-09 19:10:03.683517 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.683521 | controller | - ' ' 2026-03-09 19:10:03.683526 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.683530 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.683534 | controller | | base64 -d)' 2026-03-09 19:10:03.683539 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.683543 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.683547 | controller | - ' exit 1' 2026-03-09 19:10:03.683551 | controller | - ' fi' 2026-03-09 19:10:03.683556 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.683560 | controller | - ' ' 2026-03-09 19:10:03.683564 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.683568 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.683573 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.683577 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.683581 | controller | --timeout=300s' 2026-03-09 19:10:03.683585 | controller | - ' ' 2026-03-09 19:10:03.683590 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.683594 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.683605 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.683611 | controller | - ' ' 2026-03-09 19:10:03.683617 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683621 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683626 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683630 | controller | - ' ' 2026-03-09 19:10:03.683634 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.683639 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.683643 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.683647 | controller | available"' 2026-03-09 19:10:03.683652 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.683656 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.683660 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.683664 | controller | - ' exit 1' 2026-03-09 19:10:03.683669 | controller | - ' fi' 2026-03-09 19:10:03.683673 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.683677 | controller | - ' ' 2026-03-09 19:10:03.683682 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.683686 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.683690 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683695 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.683700 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683705 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.683709 | controller | -- \' 2026-03-09 19:10:03.683713 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683717 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.683722 | controller | - ' logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.683726 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.683730 | controller | - ' logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.683735 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.683752 | controller | found' 2026-03-09 19:10:03.683759 | controller | - ' logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.683763 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.683767 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.683772 | controller | - ' ' 2026-03-09 19:10:03.683776 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.683780 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.683785 | controller | - ' ' 2026-03-09 19:10:03.683789 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.683793 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.683797 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.683802 | controller | - ' exit 1' 2026-03-09 19:10:03.683806 | controller | - ' fi' 2026-03-09 19:10:03.683810 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.683817 | controller | - ' ' 2026-03-09 19:10:03.683821 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.683826 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.683830 | controller | | base64 -d)' 2026-03-09 19:10:03.683834 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.683839 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.683843 | controller | - ' exit 1' 2026-03-09 19:10:03.683847 | controller | - ' fi' 2026-03-09 19:10:03.683851 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.683856 | controller | - ' ' 2026-03-09 19:10:03.683860 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.683864 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.683869 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.683873 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.683877 | controller | --timeout=300s' 2026-03-09 19:10:03.683881 | controller | - ' ' 2026-03-09 19:10:03.683886 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.683890 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.683894 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.683898 | controller | - ' ' 2026-03-09 19:10:03.683910 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683915 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683919 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.683923 | controller | - ' ' 2026-03-09 19:10:03.683928 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.683932 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.683936 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.683941 | controller | available"' 2026-03-09 19:10:03.683945 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.683949 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.683953 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.683958 | controller | - ' exit 1' 2026-03-09 19:10:03.683962 | controller | - ' fi' 2026-03-09 19:10:03.683966 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.683971 | controller | - ' ' 2026-03-09 19:10:03.683975 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.683979 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.683983 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683987 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.683992 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.683996 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.684000 | controller | -- \' 2026-03-09 19:10:03.684005 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684013 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.684020 | controller | - ' logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.684026 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.684032 | controller | - ' logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.684037 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.684042 | controller | found' 2026-03-09 19:10:03.684046 | controller | - ' logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.684050 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.684054 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.684059 | controller | - ' ' 2026-03-09 19:10:03.684063 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.684067 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.684072 | controller | - ' ' 2026-03-09 19:10:03.684076 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.684080 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.684084 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.684089 | controller | - ' exit 1' 2026-03-09 19:10:03.684093 | controller | - ' fi' 2026-03-09 19:10:03.684097 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.684104 | controller | - ' ' 2026-03-09 19:10:03.684108 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.684112 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.684117 | controller | | base64 -d)' 2026-03-09 19:10:03.684121 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.684125 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.684130 | controller | - ' exit 1' 2026-03-09 19:10:03.684134 | controller | - ' fi' 2026-03-09 19:10:03.684138 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.684142 | controller | - ' ' 2026-03-09 19:10:03.684163 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.684168 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.684172 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.684176 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.684181 | controller | --timeout=300s' 2026-03-09 19:10:03.684185 | controller | - ' ' 2026-03-09 19:10:03.684189 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.684193 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.684198 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.684202 | controller | - ' ' 2026-03-09 19:10:03.684214 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684219 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684223 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684227 | controller | - ' ' 2026-03-09 19:10:03.684232 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.684239 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.684243 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.684247 | controller | available"' 2026-03-09 19:10:03.684252 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.684256 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.684260 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.684264 | controller | - ' exit 1' 2026-03-09 19:10:03.684269 | controller | - ' fi' 2026-03-09 19:10:03.684273 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.684277 | controller | - ' ' 2026-03-09 19:10:03.684282 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.684286 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.684290 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684294 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.684299 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684303 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.684307 | controller | -- \' 2026-03-09 19:10:03.684311 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684316 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.684320 | controller | - ' logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.684324 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.684329 | controller | - ' logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.684333 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.684337 | controller | found' 2026-03-09 19:10:03.684341 | controller | - ' logger.go:42: 18:51:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.684346 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.684350 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.684354 | controller | - ' ' 2026-03-09 19:10:03.684359 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.684363 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.684367 | controller | - ' ' 2026-03-09 19:10:03.684371 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.684376 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.684380 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.684384 | controller | - ' exit 1' 2026-03-09 19:10:03.684388 | controller | - ' fi' 2026-03-09 19:10:03.684393 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.684397 | controller | - ' ' 2026-03-09 19:10:03.684401 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.684406 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.684410 | controller | | base64 -d)' 2026-03-09 19:10:03.684414 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.684418 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.684423 | controller | - ' exit 1' 2026-03-09 19:10:03.684429 | controller | - ' fi' 2026-03-09 19:10:03.684433 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.684437 | controller | - ' ' 2026-03-09 19:10:03.684442 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.684446 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.684450 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.684454 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.684459 | controller | --timeout=300s' 2026-03-09 19:10:03.684463 | controller | - ' ' 2026-03-09 19:10:03.684467 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.684472 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.684476 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.684480 | controller | - ' ' 2026-03-09 19:10:03.684486 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684498 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684503 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684510 | controller | - ' ' 2026-03-09 19:10:03.684514 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.684519 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.684523 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.684527 | controller | available"' 2026-03-09 19:10:03.684532 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.684538 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.684542 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.684546 | controller | - ' exit 1' 2026-03-09 19:10:03.684550 | controller | - ' fi' 2026-03-09 19:10:03.684555 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.684559 | controller | - ' ' 2026-03-09 19:10:03.684563 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.684567 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.684572 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684576 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.684580 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684584 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.684589 | controller | -- \' 2026-03-09 19:10:03.684593 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684597 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.684601 | controller | - ' logger.go:42: 18:51:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.684606 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.684611 | controller | - ' logger.go:42: 18:51:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.684616 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.684622 | controller | found' 2026-03-09 19:10:03.684626 | controller | - ' logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.684631 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.684635 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.684639 | controller | - ' ' 2026-03-09 19:10:03.684644 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.684648 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.684652 | controller | - ' ' 2026-03-09 19:10:03.684657 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.684661 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.684665 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.684669 | controller | - ' exit 1' 2026-03-09 19:10:03.684674 | controller | - ' fi' 2026-03-09 19:10:03.684678 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.684682 | controller | - ' ' 2026-03-09 19:10:03.684687 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.684691 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.684695 | controller | | base64 -d)' 2026-03-09 19:10:03.684699 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.684704 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.684708 | controller | - ' exit 1' 2026-03-09 19:10:03.684714 | controller | - ' fi' 2026-03-09 19:10:03.684718 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.684722 | controller | - ' ' 2026-03-09 19:10:03.684727 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.684731 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.684737 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.684741 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.684745 | controller | --timeout=300s' 2026-03-09 19:10:03.684749 | controller | - ' ' 2026-03-09 19:10:03.684754 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.684758 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.684762 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.684766 | controller | - ' ' 2026-03-09 19:10:03.684771 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684784 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684790 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.684796 | controller | - ' ' 2026-03-09 19:10:03.684802 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.684808 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.684814 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.684819 | controller | available"' 2026-03-09 19:10:03.684825 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.684831 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.684837 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.684846 | controller | - ' exit 1' 2026-03-09 19:10:03.684852 | controller | - ' fi' 2026-03-09 19:10:03.684856 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.684861 | controller | - ' ' 2026-03-09 19:10:03.684865 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.684869 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.684874 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684878 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.684882 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684886 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.684891 | controller | -- \' 2026-03-09 19:10:03.684895 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.684899 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.684903 | controller | - ' logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.684908 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.684912 | controller | - ' logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.684916 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.684921 | controller | found' 2026-03-09 19:10:03.684925 | controller | - ' logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.684929 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.684933 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.684938 | controller | - ' ' 2026-03-09 19:10:03.684942 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.684946 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.684950 | controller | - ' ' 2026-03-09 19:10:03.684971 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.684976 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.684980 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.684984 | controller | - ' exit 1' 2026-03-09 19:10:03.684989 | controller | - ' fi' 2026-03-09 19:10:03.684993 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.684997 | controller | - ' ' 2026-03-09 19:10:03.685001 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.685007 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.685013 | controller | | base64 -d)' 2026-03-09 19:10:03.685018 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.685024 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.685030 | controller | - ' exit 1' 2026-03-09 19:10:03.685036 | controller | - ' fi' 2026-03-09 19:10:03.685042 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.685048 | controller | - ' ' 2026-03-09 19:10:03.685054 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.685059 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.685064 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.685068 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.685075 | controller | --timeout=300s' 2026-03-09 19:10:03.685079 | controller | - ' ' 2026-03-09 19:10:03.685083 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.685088 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.685092 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.685096 | controller | - ' ' 2026-03-09 19:10:03.685100 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685113 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685118 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685122 | controller | - ' ' 2026-03-09 19:10:03.685126 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.685131 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.685135 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.685139 | controller | available"' 2026-03-09 19:10:03.685143 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.685162 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.685167 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.685171 | controller | - ' exit 1' 2026-03-09 19:10:03.685175 | controller | - ' fi' 2026-03-09 19:10:03.685180 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.685184 | controller | - ' ' 2026-03-09 19:10:03.685188 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.685192 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.685197 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685201 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.685205 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685209 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.685214 | controller | -- \' 2026-03-09 19:10:03.685218 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685222 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.685226 | controller | - ' logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.685231 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.685235 | controller | - ' logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.685239 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.685243 | controller | found' 2026-03-09 19:10:03.685250 | controller | - ' logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.685254 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.685258 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.685262 | controller | - ' ' 2026-03-09 19:10:03.685267 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.685271 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.685277 | controller | - ' ' 2026-03-09 19:10:03.685282 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.685286 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.685290 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.685294 | controller | - ' exit 1' 2026-03-09 19:10:03.685299 | controller | - ' fi' 2026-03-09 19:10:03.685303 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.685307 | controller | - ' ' 2026-03-09 19:10:03.685312 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.685316 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.685320 | controller | | base64 -d)' 2026-03-09 19:10:03.685324 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.685329 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.685333 | controller | - ' exit 1' 2026-03-09 19:10:03.685337 | controller | - ' fi' 2026-03-09 19:10:03.685341 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.685346 | controller | - ' ' 2026-03-09 19:10:03.685350 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.685354 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.685358 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.685365 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.685369 | controller | --timeout=300s' 2026-03-09 19:10:03.685373 | controller | - ' ' 2026-03-09 19:10:03.685377 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.685382 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.685386 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.685390 | controller | - ' ' 2026-03-09 19:10:03.685394 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685399 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685411 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685415 | controller | - ' ' 2026-03-09 19:10:03.685420 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.685424 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.685428 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.685433 | controller | available"' 2026-03-09 19:10:03.685437 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.685441 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.685445 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.685450 | controller | - ' exit 1' 2026-03-09 19:10:03.685454 | controller | - ' fi' 2026-03-09 19:10:03.685458 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.685463 | controller | - ' ' 2026-03-09 19:10:03.685469 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.685474 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.685479 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685485 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.685489 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685493 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.685498 | controller | -- \' 2026-03-09 19:10:03.685502 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685506 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.685510 | controller | - ' logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.685515 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.685519 | controller | - ' logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.685523 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.685528 | controller | found' 2026-03-09 19:10:03.685532 | controller | - ' logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.685536 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.685540 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.685545 | controller | - ' ' 2026-03-09 19:10:03.685549 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.685553 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.685558 | controller | - ' ' 2026-03-09 19:10:03.685562 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.685566 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.685570 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.685575 | controller | - ' exit 1' 2026-03-09 19:10:03.685579 | controller | - ' fi' 2026-03-09 19:10:03.685583 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.685588 | controller | - ' ' 2026-03-09 19:10:03.685592 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.685596 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.685600 | controller | | base64 -d)' 2026-03-09 19:10:03.685605 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.685609 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.685613 | controller | - ' exit 1' 2026-03-09 19:10:03.685617 | controller | - ' fi' 2026-03-09 19:10:03.685622 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.685626 | controller | - ' ' 2026-03-09 19:10:03.685630 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.685635 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.685639 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.685643 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.685647 | controller | --timeout=300s' 2026-03-09 19:10:03.685652 | controller | - ' ' 2026-03-09 19:10:03.685656 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.685660 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.685665 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.685669 | controller | - ' ' 2026-03-09 19:10:03.685675 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685679 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685691 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685695 | controller | - ' ' 2026-03-09 19:10:03.685699 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.685704 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.685708 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.685712 | controller | available"' 2026-03-09 19:10:03.685717 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.685721 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.685725 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.685729 | controller | - ' exit 1' 2026-03-09 19:10:03.685734 | controller | - ' fi' 2026-03-09 19:10:03.685738 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.685742 | controller | - ' ' 2026-03-09 19:10:03.685746 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.685751 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.685755 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685759 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.685764 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685768 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.685774 | controller | -- \' 2026-03-09 19:10:03.685778 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.685782 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.685787 | controller | - ' logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.685791 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.685795 | controller | - ' logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.685799 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.685804 | controller | found' 2026-03-09 19:10:03.685809 | controller | - ' logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.685814 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.685818 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.685822 | controller | - ' ' 2026-03-09 19:10:03.685826 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.685831 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.685835 | controller | - ' ' 2026-03-09 19:10:03.685841 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.685845 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.685849 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.685854 | controller | - ' exit 1' 2026-03-09 19:10:03.685858 | controller | - ' fi' 2026-03-09 19:10:03.685862 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.685867 | controller | - ' ' 2026-03-09 19:10:03.685876 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.685880 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.685886 | controller | | base64 -d)' 2026-03-09 19:10:03.685892 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.685900 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.685906 | controller | - ' exit 1' 2026-03-09 19:10:03.685913 | controller | - ' fi' 2026-03-09 19:10:03.685919 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.685924 | controller | - ' ' 2026-03-09 19:10:03.685930 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.685936 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.685941 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.685947 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.685952 | controller | --timeout=300s' 2026-03-09 19:10:03.685958 | controller | - ' ' 2026-03-09 19:10:03.685963 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.685969 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.685975 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.685981 | controller | - ' ' 2026-03-09 19:10:03.685987 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685991 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.685996 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686002 | controller | - ' ' 2026-03-09 19:10:03.686021 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.686029 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.686035 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.686041 | controller | available"' 2026-03-09 19:10:03.686046 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.686052 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.686057 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.686063 | controller | - ' exit 1' 2026-03-09 19:10:03.686069 | controller | - ' fi' 2026-03-09 19:10:03.686075 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.686080 | controller | - ' ' 2026-03-09 19:10:03.686086 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.686092 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.686098 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686104 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.686109 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686114 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.686118 | controller | -- \' 2026-03-09 19:10:03.686124 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686131 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.686136 | controller | - ' logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.686140 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.686158 | controller | - ' logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.686165 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.686169 | controller | found' 2026-03-09 19:10:03.686175 | controller | - ' logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.686179 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.686184 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.686188 | controller | - ' ' 2026-03-09 19:10:03.686192 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.686196 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.686201 | controller | - ' ' 2026-03-09 19:10:03.686205 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.686209 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.686213 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.686218 | controller | - ' exit 1' 2026-03-09 19:10:03.686222 | controller | - ' fi' 2026-03-09 19:10:03.686226 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.686230 | controller | - ' ' 2026-03-09 19:10:03.686235 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.686239 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.686243 | controller | | base64 -d)' 2026-03-09 19:10:03.686247 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.686252 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.686256 | controller | - ' exit 1' 2026-03-09 19:10:03.686260 | controller | - ' fi' 2026-03-09 19:10:03.686264 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.686269 | controller | - ' ' 2026-03-09 19:10:03.686273 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.686277 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.686281 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.686286 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.686290 | controller | --timeout=300s' 2026-03-09 19:10:03.686294 | controller | - ' ' 2026-03-09 19:10:03.686298 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.686303 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.686307 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.686311 | controller | - ' ' 2026-03-09 19:10:03.686316 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686320 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686324 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686328 | controller | - ' ' 2026-03-09 19:10:03.686341 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.686348 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.686352 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.686357 | controller | available"' 2026-03-09 19:10:03.686361 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.686365 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.686370 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.686374 | controller | - ' exit 1' 2026-03-09 19:10:03.686378 | controller | - ' fi' 2026-03-09 19:10:03.686382 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.686387 | controller | - ' ' 2026-03-09 19:10:03.686391 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.686395 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.686400 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686404 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.686408 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686412 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.686417 | controller | -- \' 2026-03-09 19:10:03.686421 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686425 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.686430 | controller | - ' logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.686434 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.686438 | controller | - ' logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.686442 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.686447 | controller | found' 2026-03-09 19:10:03.686451 | controller | - ' logger.go:42: 18:51:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.686455 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.686459 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.686464 | controller | - ' ' 2026-03-09 19:10:03.686468 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.686472 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.686477 | controller | - ' ' 2026-03-09 19:10:03.686481 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.686485 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.686489 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.686494 | controller | - ' exit 1' 2026-03-09 19:10:03.686498 | controller | - ' fi' 2026-03-09 19:10:03.686502 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.686506 | controller | - ' ' 2026-03-09 19:10:03.686511 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.686515 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.686519 | controller | | base64 -d)' 2026-03-09 19:10:03.686523 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.686528 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.686532 | controller | - ' exit 1' 2026-03-09 19:10:03.686536 | controller | - ' fi' 2026-03-09 19:10:03.686542 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.686546 | controller | - ' ' 2026-03-09 19:10:03.686551 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.686555 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.686559 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.686564 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.686568 | controller | --timeout=300s' 2026-03-09 19:10:03.686572 | controller | - ' ' 2026-03-09 19:10:03.686576 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.686582 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.686588 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.686593 | controller | - ' ' 2026-03-09 19:10:03.686597 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686602 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686607 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686612 | controller | - ' ' 2026-03-09 19:10:03.686616 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.686630 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.686635 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.686639 | controller | available"' 2026-03-09 19:10:03.686643 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.686648 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.686652 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.686656 | controller | - ' exit 1' 2026-03-09 19:10:03.686660 | controller | - ' fi' 2026-03-09 19:10:03.686665 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.686669 | controller | - ' ' 2026-03-09 19:10:03.686673 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.686677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.686682 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686686 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.686690 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686694 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.686699 | controller | -- \' 2026-03-09 19:10:03.686703 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686707 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.686712 | controller | - ' logger.go:42: 18:51:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.686716 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.686720 | controller | - ' logger.go:42: 18:51:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.686724 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.686730 | controller | found' 2026-03-09 19:10:03.686737 | controller | - ' logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.686742 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.686746 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.686750 | controller | - ' ' 2026-03-09 19:10:03.686755 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.686759 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.686763 | controller | - ' ' 2026-03-09 19:10:03.686767 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.686772 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.686776 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.686780 | controller | - ' exit 1' 2026-03-09 19:10:03.686784 | controller | - ' fi' 2026-03-09 19:10:03.686789 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.686793 | controller | - ' ' 2026-03-09 19:10:03.686797 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.686801 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.686806 | controller | | base64 -d)' 2026-03-09 19:10:03.686810 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.686814 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.686818 | controller | - ' exit 1' 2026-03-09 19:10:03.686823 | controller | - ' fi' 2026-03-09 19:10:03.686827 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.686831 | controller | - ' ' 2026-03-09 19:10:03.686835 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.686840 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.686844 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.686848 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.686852 | controller | --timeout=300s' 2026-03-09 19:10:03.686857 | controller | - ' ' 2026-03-09 19:10:03.686861 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.686865 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.686869 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.686873 | controller | - ' ' 2026-03-09 19:10:03.686878 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686882 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686888 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.686892 | controller | - ' ' 2026-03-09 19:10:03.686896 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.686901 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.686911 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.686916 | controller | available"' 2026-03-09 19:10:03.686922 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.686926 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.686931 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.686935 | controller | - ' exit 1' 2026-03-09 19:10:03.686941 | controller | - ' fi' 2026-03-09 19:10:03.686945 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.686950 | controller | - ' ' 2026-03-09 19:10:03.686954 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.686958 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.686962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686967 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.686973 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686977 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.686981 | controller | -- \' 2026-03-09 19:10:03.686986 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.686990 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.686994 | controller | - ' logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.686999 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.687003 | controller | - ' logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.687007 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.687011 | controller | found' 2026-03-09 19:10:03.687017 | controller | - ' logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.687022 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.687028 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.687033 | controller | - ' ' 2026-03-09 19:10:03.687037 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.687041 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.687045 | controller | - ' ' 2026-03-09 19:10:03.687050 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.687054 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.687058 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.687062 | controller | - ' exit 1' 2026-03-09 19:10:03.687067 | controller | - ' fi' 2026-03-09 19:10:03.687071 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.687075 | controller | - ' ' 2026-03-09 19:10:03.687079 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.687084 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.687088 | controller | | base64 -d)' 2026-03-09 19:10:03.687092 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.687096 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.687101 | controller | - ' exit 1' 2026-03-09 19:10:03.687105 | controller | - ' fi' 2026-03-09 19:10:03.687109 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.687113 | controller | - ' ' 2026-03-09 19:10:03.687118 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.687122 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.687126 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.687130 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.687134 | controller | --timeout=300s' 2026-03-09 19:10:03.687141 | controller | - ' ' 2026-03-09 19:10:03.687163 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.687170 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.687174 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.687179 | controller | - ' ' 2026-03-09 19:10:03.687183 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687187 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687191 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687196 | controller | - ' ' 2026-03-09 19:10:03.687200 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.687204 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.687216 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.687221 | controller | available"' 2026-03-09 19:10:03.687225 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.687232 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.687236 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.687241 | controller | - ' exit 1' 2026-03-09 19:10:03.687245 | controller | - ' fi' 2026-03-09 19:10:03.687252 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.687256 | controller | - ' ' 2026-03-09 19:10:03.687261 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.687265 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.687269 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687274 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.687278 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687282 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.687286 | controller | -- \' 2026-03-09 19:10:03.687291 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687295 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.687301 | controller | - ' logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.687305 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.687309 | controller | - ' logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.687314 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.687318 | controller | found' 2026-03-09 19:10:03.687322 | controller | - ' logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.687326 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.687331 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.687335 | controller | - ' ' 2026-03-09 19:10:03.687339 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.687343 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.687348 | controller | - ' ' 2026-03-09 19:10:03.687354 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.687358 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.687363 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.687367 | controller | - ' exit 1' 2026-03-09 19:10:03.687371 | controller | - ' fi' 2026-03-09 19:10:03.687376 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.687380 | controller | - ' ' 2026-03-09 19:10:03.687384 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.687389 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.687395 | controller | | base64 -d)' 2026-03-09 19:10:03.687400 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.687404 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.687408 | controller | - ' exit 1' 2026-03-09 19:10:03.687413 | controller | - ' fi' 2026-03-09 19:10:03.687417 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.687421 | controller | - ' ' 2026-03-09 19:10:03.687426 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.687430 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.687434 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.687438 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.687443 | controller | --timeout=300s' 2026-03-09 19:10:03.687449 | controller | - ' ' 2026-03-09 19:10:03.687453 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.687457 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.687462 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.687466 | controller | - ' ' 2026-03-09 19:10:03.687470 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687474 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687479 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687483 | controller | - ' ' 2026-03-09 19:10:03.687487 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.687491 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.687496 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.687500 | controller | available"' 2026-03-09 19:10:03.687512 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.687516 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.687520 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.687525 | controller | - ' exit 1' 2026-03-09 19:10:03.687529 | controller | - ' fi' 2026-03-09 19:10:03.687533 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.687538 | controller | - ' ' 2026-03-09 19:10:03.687542 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.687546 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.687550 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687557 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.687561 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687565 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.687570 | controller | -- \' 2026-03-09 19:10:03.687574 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687578 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.687583 | controller | - ' logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.687587 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.687591 | controller | - ' logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.687596 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.687600 | controller | found' 2026-03-09 19:10:03.687605 | controller | - ' logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.687610 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.687614 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.687619 | controller | - ' ' 2026-03-09 19:10:03.687623 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.687627 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.687631 | controller | - ' ' 2026-03-09 19:10:03.687636 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.687642 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.687647 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.687651 | controller | - ' exit 1' 2026-03-09 19:10:03.687656 | controller | - ' fi' 2026-03-09 19:10:03.687660 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.687664 | controller | - ' ' 2026-03-09 19:10:03.687668 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.687673 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.687677 | controller | | base64 -d)' 2026-03-09 19:10:03.687681 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.687686 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.687690 | controller | - ' exit 1' 2026-03-09 19:10:03.687694 | controller | - ' fi' 2026-03-09 19:10:03.687698 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.687703 | controller | - ' ' 2026-03-09 19:10:03.687707 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.687711 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.687715 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.687720 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.687724 | controller | --timeout=300s' 2026-03-09 19:10:03.687728 | controller | - ' ' 2026-03-09 19:10:03.687732 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.687750 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.687758 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.687764 | controller | - ' ' 2026-03-09 19:10:03.687768 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687775 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687779 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.687783 | controller | - ' ' 2026-03-09 19:10:03.687788 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.687792 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.687796 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.687800 | controller | available"' 2026-03-09 19:10:03.687805 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.687817 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.687822 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.687826 | controller | - ' exit 1' 2026-03-09 19:10:03.687830 | controller | - ' fi' 2026-03-09 19:10:03.687835 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.687839 | controller | - ' ' 2026-03-09 19:10:03.687845 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.687850 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.687855 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687860 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.687865 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687869 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.687874 | controller | -- \' 2026-03-09 19:10:03.687878 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.687882 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.687888 | controller | - ' logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.687893 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.687897 | controller | - ' logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.687901 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.687906 | controller | found' 2026-03-09 19:10:03.687910 | controller | - ' logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.687914 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.687919 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.687923 | controller | - ' ' 2026-03-09 19:10:03.687927 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.687932 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.687936 | controller | - ' ' 2026-03-09 19:10:03.687940 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.687944 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.687949 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.687953 | controller | - ' exit 1' 2026-03-09 19:10:03.687957 | controller | - ' fi' 2026-03-09 19:10:03.687962 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.687966 | controller | - ' ' 2026-03-09 19:10:03.687970 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.687976 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.687980 | controller | | base64 -d)' 2026-03-09 19:10:03.687985 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.687989 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.687993 | controller | - ' exit 1' 2026-03-09 19:10:03.687998 | controller | - ' fi' 2026-03-09 19:10:03.688002 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.688006 | controller | - ' ' 2026-03-09 19:10:03.688011 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.688015 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.688021 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.688025 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.688029 | controller | --timeout=300s' 2026-03-09 19:10:03.688034 | controller | - ' ' 2026-03-09 19:10:03.688038 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.688042 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.688047 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.688051 | controller | - ' ' 2026-03-09 19:10:03.688055 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688059 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688065 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688070 | controller | - ' ' 2026-03-09 19:10:03.688075 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.688079 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.688083 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.688088 | controller | available"' 2026-03-09 19:10:03.688092 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.688103 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.688108 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.688112 | controller | - ' exit 1' 2026-03-09 19:10:03.688116 | controller | - ' fi' 2026-03-09 19:10:03.688121 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.688125 | controller | - ' ' 2026-03-09 19:10:03.688129 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.688133 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.688138 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688142 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.688162 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688166 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.688170 | controller | -- \' 2026-03-09 19:10:03.688175 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688179 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.688186 | controller | - ' logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.688191 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.688195 | controller | - ' logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.688199 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.688204 | controller | found' 2026-03-09 19:10:03.688208 | controller | - ' logger.go:42: 18:51:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.688212 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.688217 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.688223 | controller | - ' ' 2026-03-09 19:10:03.688228 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.688232 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.688236 | controller | - ' ' 2026-03-09 19:10:03.688241 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.688245 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.688249 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.688254 | controller | - ' exit 1' 2026-03-09 19:10:03.688258 | controller | - ' fi' 2026-03-09 19:10:03.688262 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.688267 | controller | - ' ' 2026-03-09 19:10:03.688271 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.688275 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.688280 | controller | | base64 -d)' 2026-03-09 19:10:03.688284 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.688288 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.688292 | controller | - ' exit 1' 2026-03-09 19:10:03.688297 | controller | - ' fi' 2026-03-09 19:10:03.688301 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.688305 | controller | - ' ' 2026-03-09 19:10:03.688310 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.688314 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.688318 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.688322 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.688327 | controller | --timeout=300s' 2026-03-09 19:10:03.688331 | controller | - ' ' 2026-03-09 19:10:03.688335 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.688340 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.688344 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.688348 | controller | - ' ' 2026-03-09 19:10:03.688353 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688357 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688361 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688366 | controller | - ' ' 2026-03-09 19:10:03.688370 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.688374 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.688380 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.688384 | controller | available"' 2026-03-09 19:10:03.688389 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.688393 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.688405 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.688410 | controller | - ' exit 1' 2026-03-09 19:10:03.688415 | controller | - ' fi' 2026-03-09 19:10:03.688419 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.688423 | controller | - ' ' 2026-03-09 19:10:03.688428 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.688432 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.688436 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688442 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.688447 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688451 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.688455 | controller | -- \' 2026-03-09 19:10:03.688460 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688464 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.688468 | controller | - ' logger.go:42: 18:51:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.688473 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.688477 | controller | - ' logger.go:42: 18:51:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.688481 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.688486 | controller | found' 2026-03-09 19:10:03.688490 | controller | - ' logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.688494 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.688499 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.688503 | controller | - ' ' 2026-03-09 19:10:03.688507 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.688511 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.688516 | controller | - ' ' 2026-03-09 19:10:03.688520 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.688524 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.688529 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.688533 | controller | - ' exit 1' 2026-03-09 19:10:03.688537 | controller | - ' fi' 2026-03-09 19:10:03.688541 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.688546 | controller | - ' ' 2026-03-09 19:10:03.688552 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.688556 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.688560 | controller | | base64 -d)' 2026-03-09 19:10:03.688565 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.688569 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.688573 | controller | - ' exit 1' 2026-03-09 19:10:03.688578 | controller | - ' fi' 2026-03-09 19:10:03.688582 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.688594 | controller | - ' ' 2026-03-09 19:10:03.688598 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.688602 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.688607 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.688611 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.688615 | controller | --timeout=300s' 2026-03-09 19:10:03.688620 | controller | - ' ' 2026-03-09 19:10:03.688624 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.688628 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.688632 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.688637 | controller | - ' ' 2026-03-09 19:10:03.688643 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688647 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688651 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688656 | controller | - ' ' 2026-03-09 19:10:03.688660 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.688664 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.688668 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.688673 | controller | available"' 2026-03-09 19:10:03.688677 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.688681 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.688686 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.688697 | controller | - ' exit 1' 2026-03-09 19:10:03.688701 | controller | - ' fi' 2026-03-09 19:10:03.688706 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.688710 | controller | - ' ' 2026-03-09 19:10:03.688714 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.688719 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.688723 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688727 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.688731 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688736 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.688740 | controller | -- \' 2026-03-09 19:10:03.688744 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.688749 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.688753 | controller | - ' logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.688757 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.688761 | controller | - ' logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.688766 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.688770 | controller | found' 2026-03-09 19:10:03.688774 | controller | - ' logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.688781 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.688785 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.688789 | controller | - ' ' 2026-03-09 19:10:03.688794 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.688798 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.688802 | controller | - ' ' 2026-03-09 19:10:03.688807 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.688811 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.688815 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.688819 | controller | - ' exit 1' 2026-03-09 19:10:03.688824 | controller | - ' fi' 2026-03-09 19:10:03.688828 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.688832 | controller | - ' ' 2026-03-09 19:10:03.688837 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.688841 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.688845 | controller | | base64 -d)' 2026-03-09 19:10:03.688850 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.688854 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.688858 | controller | - ' exit 1' 2026-03-09 19:10:03.688863 | controller | - ' fi' 2026-03-09 19:10:03.688867 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.688871 | controller | - ' ' 2026-03-09 19:10:03.688875 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.688880 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.688884 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.688888 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.688893 | controller | --timeout=300s' 2026-03-09 19:10:03.688897 | controller | - ' ' 2026-03-09 19:10:03.688901 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.688905 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.688910 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.688914 | controller | - ' ' 2026-03-09 19:10:03.688918 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688923 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688927 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.688931 | controller | - ' ' 2026-03-09 19:10:03.688936 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.688942 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.688946 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.688950 | controller | available"' 2026-03-09 19:10:03.688955 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.688959 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.688963 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.688967 | controller | - ' exit 1' 2026-03-09 19:10:03.688972 | controller | - ' fi' 2026-03-09 19:10:03.688985 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.688989 | controller | - ' ' 2026-03-09 19:10:03.688994 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.688998 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.689002 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689007 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.689011 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689015 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.689019 | controller | -- \' 2026-03-09 19:10:03.689024 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689030 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.689034 | controller | - ' logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.689038 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.689043 | controller | - ' logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.689047 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.689051 | controller | found' 2026-03-09 19:10:03.689055 | controller | - ' logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.689060 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.689064 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.689068 | controller | - ' ' 2026-03-09 19:10:03.689072 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.689077 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.689081 | controller | - ' ' 2026-03-09 19:10:03.689085 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.689089 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.689094 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.689098 | controller | - ' exit 1' 2026-03-09 19:10:03.689102 | controller | - ' fi' 2026-03-09 19:10:03.689108 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.689113 | controller | - ' ' 2026-03-09 19:10:03.689117 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.689121 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.689126 | controller | | base64 -d)' 2026-03-09 19:10:03.689130 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.689134 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.689138 | controller | - ' exit 1' 2026-03-09 19:10:03.689143 | controller | - ' fi' 2026-03-09 19:10:03.689161 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.689166 | controller | - ' ' 2026-03-09 19:10:03.689170 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.689174 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.689178 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.689183 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.689187 | controller | --timeout=300s' 2026-03-09 19:10:03.689191 | controller | - ' ' 2026-03-09 19:10:03.689198 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.689202 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.689206 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.689210 | controller | - ' ' 2026-03-09 19:10:03.689215 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689219 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689223 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689227 | controller | - ' ' 2026-03-09 19:10:03.689232 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.689236 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.689240 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.689244 | controller | available"' 2026-03-09 19:10:03.689249 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.689253 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.689257 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.689262 | controller | - ' exit 1' 2026-03-09 19:10:03.689266 | controller | - ' fi' 2026-03-09 19:10:03.689277 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.689282 | controller | - ' ' 2026-03-09 19:10:03.689286 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.689290 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.689295 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689299 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.689303 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689308 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.689312 | controller | -- \' 2026-03-09 19:10:03.689316 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689320 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.689325 | controller | - ' logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.689329 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.689333 | controller | - ' logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.689337 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.689342 | controller | found' 2026-03-09 19:10:03.689346 | controller | - ' logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.689350 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.689354 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.689359 | controller | - ' ' 2026-03-09 19:10:03.689363 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.689367 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.689374 | controller | - ' ' 2026-03-09 19:10:03.689378 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.689384 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.689388 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.689393 | controller | - ' exit 1' 2026-03-09 19:10:03.689397 | controller | - ' fi' 2026-03-09 19:10:03.689401 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.689406 | controller | - ' ' 2026-03-09 19:10:03.689410 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.689414 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.689419 | controller | | base64 -d)' 2026-03-09 19:10:03.689423 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.689427 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.689431 | controller | - ' exit 1' 2026-03-09 19:10:03.689436 | controller | - ' fi' 2026-03-09 19:10:03.689440 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.689444 | controller | - ' ' 2026-03-09 19:10:03.689449 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.689453 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.689457 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.689462 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.689466 | controller | --timeout=300s' 2026-03-09 19:10:03.689470 | controller | - ' ' 2026-03-09 19:10:03.689474 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.689479 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.689483 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.689487 | controller | - ' ' 2026-03-09 19:10:03.689491 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689496 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689500 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689504 | controller | - ' ' 2026-03-09 19:10:03.689508 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.689513 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.689517 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.689521 | controller | available"' 2026-03-09 19:10:03.689526 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.689530 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.689534 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.689538 | controller | - ' exit 1' 2026-03-09 19:10:03.689543 | controller | - ' fi' 2026-03-09 19:10:03.689547 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.689551 | controller | - ' ' 2026-03-09 19:10:03.689562 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.689566 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.689571 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689575 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.689581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689586 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.689590 | controller | -- \' 2026-03-09 19:10:03.689594 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689598 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.689603 | controller | - ' logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.689607 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.689613 | controller | - ' logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.689617 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.689622 | controller | found' 2026-03-09 19:10:03.689626 | controller | - ' logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.689630 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.689634 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.689639 | controller | - ' ' 2026-03-09 19:10:03.689643 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.689647 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.689651 | controller | - ' ' 2026-03-09 19:10:03.689656 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.689660 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.689664 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.689668 | controller | - ' exit 1' 2026-03-09 19:10:03.689673 | controller | - ' fi' 2026-03-09 19:10:03.689677 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.689681 | controller | - ' ' 2026-03-09 19:10:03.689686 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.689690 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.689694 | controller | | base64 -d)' 2026-03-09 19:10:03.689698 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.689703 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.689707 | controller | - ' exit 1' 2026-03-09 19:10:03.689711 | controller | - ' fi' 2026-03-09 19:10:03.689715 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.689720 | controller | - ' ' 2026-03-09 19:10:03.689724 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.689728 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.689732 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.689737 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.689741 | controller | --timeout=300s' 2026-03-09 19:10:03.689745 | controller | - ' ' 2026-03-09 19:10:03.689749 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.689754 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.689758 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.689762 | controller | - ' ' 2026-03-09 19:10:03.689767 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689772 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689777 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.689781 | controller | - ' ' 2026-03-09 19:10:03.689785 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.689790 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.689794 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.689798 | controller | available"' 2026-03-09 19:10:03.689802 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.689809 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.689814 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.689818 | controller | - ' exit 1' 2026-03-09 19:10:03.689822 | controller | - ' fi' 2026-03-09 19:10:03.689826 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.689831 | controller | - ' ' 2026-03-09 19:10:03.689842 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.689846 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.689850 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689855 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.689859 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689863 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.689868 | controller | -- \' 2026-03-09 19:10:03.689872 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.689876 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.689880 | controller | - ' logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.689885 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.689889 | controller | - ' logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.689893 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.689897 | controller | found' 2026-03-09 19:10:03.689902 | controller | - ' logger.go:42: 18:51:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.689906 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.689910 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.689915 | controller | - ' ' 2026-03-09 19:10:03.689919 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.689923 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.689927 | controller | - ' ' 2026-03-09 19:10:03.689932 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.689936 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.689940 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.689944 | controller | - ' exit 1' 2026-03-09 19:10:03.689949 | controller | - ' fi' 2026-03-09 19:10:03.689953 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.689957 | controller | - ' ' 2026-03-09 19:10:03.689961 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.689966 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.689972 | controller | | base64 -d)' 2026-03-09 19:10:03.689976 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.689988 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.689992 | controller | - ' exit 1' 2026-03-09 19:10:03.689997 | controller | - ' fi' 2026-03-09 19:10:03.690001 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.690005 | controller | - ' ' 2026-03-09 19:10:03.690010 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.690014 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.690018 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.690022 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.690027 | controller | --timeout=300s' 2026-03-09 19:10:03.690031 | controller | - ' ' 2026-03-09 19:10:03.690035 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.690040 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.690045 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.690050 | controller | - ' ' 2026-03-09 19:10:03.690054 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690058 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690062 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690067 | controller | - ' ' 2026-03-09 19:10:03.690071 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.690075 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.690080 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.690086 | controller | available"' 2026-03-09 19:10:03.690091 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.690096 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.690100 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.690104 | controller | - ' exit 1' 2026-03-09 19:10:03.690108 | controller | - ' fi' 2026-03-09 19:10:03.690115 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.690119 | controller | - ' ' 2026-03-09 19:10:03.690123 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.690135 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.690140 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690155 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.690162 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690166 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.690171 | controller | -- \' 2026-03-09 19:10:03.690176 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690180 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.690186 | controller | - ' logger.go:42: 18:51:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.690191 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.690195 | controller | - ' logger.go:42: 18:52:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.690201 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.690205 | controller | found' 2026-03-09 19:10:03.690210 | controller | - ' logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.690214 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.690218 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.690223 | controller | - ' ' 2026-03-09 19:10:03.690227 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.690231 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.690235 | controller | - ' ' 2026-03-09 19:10:03.690240 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.690244 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.690248 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.690252 | controller | - ' exit 1' 2026-03-09 19:10:03.690257 | controller | - ' fi' 2026-03-09 19:10:03.690261 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.690265 | controller | - ' ' 2026-03-09 19:10:03.690270 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.690274 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.690278 | controller | | base64 -d)' 2026-03-09 19:10:03.690282 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.690287 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.690291 | controller | - ' exit 1' 2026-03-09 19:10:03.690295 | controller | - ' fi' 2026-03-09 19:10:03.690299 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.690304 | controller | - ' ' 2026-03-09 19:10:03.690308 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.690312 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.690316 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.690321 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.690325 | controller | --timeout=300s' 2026-03-09 19:10:03.690329 | controller | - ' ' 2026-03-09 19:10:03.690334 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.690338 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.690342 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.690346 | controller | - ' ' 2026-03-09 19:10:03.690351 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690355 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690359 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690363 | controller | - ' ' 2026-03-09 19:10:03.690368 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.690372 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.690376 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.690382 | controller | available"' 2026-03-09 19:10:03.690387 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.690391 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.690395 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.690400 | controller | - ' exit 1' 2026-03-09 19:10:03.690404 | controller | - ' fi' 2026-03-09 19:10:03.690408 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.690413 | controller | - ' ' 2026-03-09 19:10:03.690417 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.690421 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.690434 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690438 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.690443 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690447 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.690451 | controller | -- \' 2026-03-09 19:10:03.690456 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690460 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.690464 | controller | - ' logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.690468 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.690473 | controller | - ' logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.690477 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.690481 | controller | found' 2026-03-09 19:10:03.690486 | controller | - ' logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.690490 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.690494 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.690498 | controller | - ' ' 2026-03-09 19:10:03.690503 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.690507 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.690511 | controller | - ' ' 2026-03-09 19:10:03.690515 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.690520 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.690524 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.690528 | controller | - ' exit 1' 2026-03-09 19:10:03.690532 | controller | - ' fi' 2026-03-09 19:10:03.690537 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.690541 | controller | - ' ' 2026-03-09 19:10:03.690545 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.690550 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.690554 | controller | | base64 -d)' 2026-03-09 19:10:03.690558 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.690562 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.690567 | controller | - ' exit 1' 2026-03-09 19:10:03.690571 | controller | - ' fi' 2026-03-09 19:10:03.690575 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.690580 | controller | - ' ' 2026-03-09 19:10:03.690584 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.690590 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.690594 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.690598 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.690603 | controller | --timeout=300s' 2026-03-09 19:10:03.690607 | controller | - ' ' 2026-03-09 19:10:03.690611 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.690616 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.690620 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.690624 | controller | - ' ' 2026-03-09 19:10:03.690628 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690633 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690637 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690641 | controller | - ' ' 2026-03-09 19:10:03.690646 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.690650 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.690654 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.690658 | controller | available"' 2026-03-09 19:10:03.690663 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.690667 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.690671 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.690676 | controller | - ' exit 1' 2026-03-09 19:10:03.690680 | controller | - ' fi' 2026-03-09 19:10:03.690684 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.690688 | controller | - ' ' 2026-03-09 19:10:03.690693 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.690697 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.690708 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690712 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.690716 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690721 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.690725 | controller | -- \' 2026-03-09 19:10:03.690729 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690734 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.690738 | controller | - ' logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.690742 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.690746 | controller | - ' logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.690751 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.690755 | controller | found' 2026-03-09 19:10:03.690762 | controller | - ' logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.690768 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.690772 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.690777 | controller | - ' ' 2026-03-09 19:10:03.690781 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.690785 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.690789 | controller | - ' ' 2026-03-09 19:10:03.690794 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.690798 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.690802 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.690807 | controller | - ' exit 1' 2026-03-09 19:10:03.690811 | controller | - ' fi' 2026-03-09 19:10:03.690815 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.690819 | controller | - ' ' 2026-03-09 19:10:03.690824 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.690828 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.690832 | controller | | base64 -d)' 2026-03-09 19:10:03.690836 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.690841 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.690845 | controller | - ' exit 1' 2026-03-09 19:10:03.690849 | controller | - ' fi' 2026-03-09 19:10:03.690854 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.690858 | controller | - ' ' 2026-03-09 19:10:03.690862 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.690866 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.690871 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.690875 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.690879 | controller | --timeout=300s' 2026-03-09 19:10:03.690883 | controller | - ' ' 2026-03-09 19:10:03.690888 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.690892 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.690896 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.690901 | controller | - ' ' 2026-03-09 19:10:03.690905 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690909 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690913 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.690918 | controller | - ' ' 2026-03-09 19:10:03.690922 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.690926 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.690930 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.690935 | controller | available"' 2026-03-09 19:10:03.690939 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.690943 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.690948 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.690952 | controller | - ' exit 1' 2026-03-09 19:10:03.690956 | controller | - ' fi' 2026-03-09 19:10:03.690960 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.690966 | controller | - ' ' 2026-03-09 19:10:03.690971 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.690975 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.690988 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.690993 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.690997 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691001 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.691006 | controller | -- \' 2026-03-09 19:10:03.691010 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691014 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.691018 | controller | - ' logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.691023 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.691027 | controller | - ' logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.691031 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.691035 | controller | found' 2026-03-09 19:10:03.691040 | controller | - ' logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.691044 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.691048 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.691052 | controller | - ' ' 2026-03-09 19:10:03.691057 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.691061 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.691065 | controller | - ' ' 2026-03-09 19:10:03.691069 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.691074 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.691078 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.691082 | controller | - ' exit 1' 2026-03-09 19:10:03.691087 | controller | - ' fi' 2026-03-09 19:10:03.691091 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.691095 | controller | - ' ' 2026-03-09 19:10:03.691101 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.691105 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.691110 | controller | | base64 -d)' 2026-03-09 19:10:03.691114 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.691118 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.691123 | controller | - ' exit 1' 2026-03-09 19:10:03.691127 | controller | - ' fi' 2026-03-09 19:10:03.691131 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.691136 | controller | - ' ' 2026-03-09 19:10:03.691140 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.691164 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.691172 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.691176 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.691181 | controller | --timeout=300s' 2026-03-09 19:10:03.691185 | controller | - ' ' 2026-03-09 19:10:03.691191 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.691198 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.691203 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.691207 | controller | - ' ' 2026-03-09 19:10:03.691211 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691216 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691220 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691224 | controller | - ' ' 2026-03-09 19:10:03.691228 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.691233 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.691237 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.691241 | controller | available"' 2026-03-09 19:10:03.691246 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.691250 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.691254 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.691259 | controller | - ' exit 1' 2026-03-09 19:10:03.691263 | controller | - ' fi' 2026-03-09 19:10:03.691267 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.691271 | controller | - ' ' 2026-03-09 19:10:03.691276 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.691280 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.691284 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691296 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.691300 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691305 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.691309 | controller | -- \' 2026-03-09 19:10:03.691313 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691318 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.691324 | controller | - ' logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.691328 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.691333 | controller | - ' logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.691337 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.691341 | controller | found' 2026-03-09 19:10:03.691345 | controller | - ' logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.691350 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.691354 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.691358 | controller | - ' ' 2026-03-09 19:10:03.691366 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.691370 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.691374 | controller | - ' ' 2026-03-09 19:10:03.691379 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.691383 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.691389 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.691394 | controller | - ' exit 1' 2026-03-09 19:10:03.691398 | controller | - ' fi' 2026-03-09 19:10:03.691402 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.691407 | controller | - ' ' 2026-03-09 19:10:03.691411 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.691416 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.691421 | controller | | base64 -d)' 2026-03-09 19:10:03.691425 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.691429 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.691433 | controller | - ' exit 1' 2026-03-09 19:10:03.691438 | controller | - ' fi' 2026-03-09 19:10:03.691442 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.691446 | controller | - ' ' 2026-03-09 19:10:03.691451 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.691455 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.691459 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.691463 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.691468 | controller | --timeout=300s' 2026-03-09 19:10:03.691474 | controller | - ' ' 2026-03-09 19:10:03.691478 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.691482 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.691486 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.691491 | controller | - ' ' 2026-03-09 19:10:03.691495 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691499 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691504 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691508 | controller | - ' ' 2026-03-09 19:10:03.691512 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.691516 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.691521 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.691525 | controller | available"' 2026-03-09 19:10:03.691529 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.691533 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.691538 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.691542 | controller | - ' exit 1' 2026-03-09 19:10:03.691546 | controller | - ' fi' 2026-03-09 19:10:03.691551 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.691555 | controller | - ' ' 2026-03-09 19:10:03.691559 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.691563 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.691568 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691572 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.691583 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691589 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.691594 | controller | -- \' 2026-03-09 19:10:03.691598 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691602 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.691607 | controller | - ' logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.691611 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.691615 | controller | - ' logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.691619 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.691624 | controller | found' 2026-03-09 19:10:03.691628 | controller | - ' logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.691632 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.691637 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.691641 | controller | - ' ' 2026-03-09 19:10:03.691645 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.691649 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.691654 | controller | - ' ' 2026-03-09 19:10:03.691658 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.691662 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.691666 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.691671 | controller | - ' exit 1' 2026-03-09 19:10:03.691675 | controller | - ' fi' 2026-03-09 19:10:03.691680 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.691684 | controller | - ' ' 2026-03-09 19:10:03.691713 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.691718 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.691723 | controller | | base64 -d)' 2026-03-09 19:10:03.691727 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.691744 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.691750 | controller | - ' exit 1' 2026-03-09 19:10:03.691756 | controller | - ' fi' 2026-03-09 19:10:03.691760 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.691767 | controller | - ' ' 2026-03-09 19:10:03.691772 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.691776 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.691780 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.691785 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.691789 | controller | --timeout=300s' 2026-03-09 19:10:03.691793 | controller | - ' ' 2026-03-09 19:10:03.691797 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.691802 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.691806 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.691810 | controller | - ' ' 2026-03-09 19:10:03.691815 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691819 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691825 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.691830 | controller | - ' ' 2026-03-09 19:10:03.691834 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.691838 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.691842 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.691847 | controller | available"' 2026-03-09 19:10:03.691851 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.691855 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.691860 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.691864 | controller | - ' exit 1' 2026-03-09 19:10:03.691868 | controller | - ' fi' 2026-03-09 19:10:03.691873 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.691877 | controller | - ' ' 2026-03-09 19:10:03.691881 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.691885 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.691890 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691894 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.691907 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691911 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.691916 | controller | -- \' 2026-03-09 19:10:03.691920 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.691924 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.691929 | controller | - ' logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.691933 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.691939 | controller | - ' logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.691944 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.691948 | controller | found' 2026-03-09 19:10:03.691952 | controller | - ' logger.go:42: 18:52:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.691957 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.691961 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.691966 | controller | - ' ' 2026-03-09 19:10:03.691970 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.691976 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.691980 | controller | - ' ' 2026-03-09 19:10:03.691984 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.691989 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.691993 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.691997 | controller | - ' exit 1' 2026-03-09 19:10:03.692002 | controller | - ' fi' 2026-03-09 19:10:03.692006 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.692010 | controller | - ' ' 2026-03-09 19:10:03.692015 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.692019 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.692025 | controller | | base64 -d)' 2026-03-09 19:10:03.692029 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.692034 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.692038 | controller | - ' exit 1' 2026-03-09 19:10:03.692042 | controller | - ' fi' 2026-03-09 19:10:03.692046 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.692051 | controller | - ' ' 2026-03-09 19:10:03.692055 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.692059 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.692064 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.692068 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.692072 | controller | --timeout=300s' 2026-03-09 19:10:03.692076 | controller | - ' ' 2026-03-09 19:10:03.692081 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.692085 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.692089 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.692094 | controller | - ' ' 2026-03-09 19:10:03.692098 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692104 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692108 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692112 | controller | - ' ' 2026-03-09 19:10:03.692117 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.692121 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.692125 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.692130 | controller | available"' 2026-03-09 19:10:03.692134 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.692138 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.692142 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.692165 | controller | - ' exit 1' 2026-03-09 19:10:03.692170 | controller | - ' fi' 2026-03-09 19:10:03.692174 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.692178 | controller | - ' ' 2026-03-09 19:10:03.692182 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.692187 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.692191 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692195 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.692207 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692211 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.692216 | controller | -- \' 2026-03-09 19:10:03.692220 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692225 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.692229 | controller | - ' logger.go:42: 18:52:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.692235 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.692240 | controller | - ' logger.go:42: 18:52:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.692244 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.692248 | controller | found' 2026-03-09 19:10:03.692252 | controller | - ' logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.692257 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.692261 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.692265 | controller | - ' ' 2026-03-09 19:10:03.692270 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.692274 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.692278 | controller | - ' ' 2026-03-09 19:10:03.692282 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.692286 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.692291 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.692295 | controller | - ' exit 1' 2026-03-09 19:10:03.692299 | controller | - ' fi' 2026-03-09 19:10:03.692304 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.692308 | controller | - ' ' 2026-03-09 19:10:03.692312 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.692317 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.692321 | controller | | base64 -d)' 2026-03-09 19:10:03.692327 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.692332 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.692336 | controller | - ' exit 1' 2026-03-09 19:10:03.692340 | controller | - ' fi' 2026-03-09 19:10:03.692344 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.692349 | controller | - ' ' 2026-03-09 19:10:03.692353 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.692357 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.692361 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.692366 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.692370 | controller | --timeout=300s' 2026-03-09 19:10:03.692374 | controller | - ' ' 2026-03-09 19:10:03.692379 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.692383 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.692387 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.692391 | controller | - ' ' 2026-03-09 19:10:03.692396 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692400 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692404 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692408 | controller | - ' ' 2026-03-09 19:10:03.692412 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.692417 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.692421 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.692427 | controller | available"' 2026-03-09 19:10:03.692432 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.692438 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.692442 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.692446 | controller | - ' exit 1' 2026-03-09 19:10:03.692451 | controller | - ' fi' 2026-03-09 19:10:03.692455 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.692459 | controller | - ' ' 2026-03-09 19:10:03.692463 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.692468 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.692472 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692476 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.692480 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692491 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.692496 | controller | -- \' 2026-03-09 19:10:03.692500 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692504 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.692509 | controller | - ' logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.692513 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.692517 | controller | - ' logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.692522 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.692526 | controller | found' 2026-03-09 19:10:03.692530 | controller | - ' logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.692534 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.692539 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.692543 | controller | - ' ' 2026-03-09 19:10:03.692547 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.692552 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.692556 | controller | - ' ' 2026-03-09 19:10:03.692563 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.692568 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.692572 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.692577 | controller | - ' exit 1' 2026-03-09 19:10:03.692581 | controller | - ' fi' 2026-03-09 19:10:03.692585 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.692589 | controller | - ' ' 2026-03-09 19:10:03.692594 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.692598 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.692602 | controller | | base64 -d)' 2026-03-09 19:10:03.692607 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.692611 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.692615 | controller | - ' exit 1' 2026-03-09 19:10:03.692619 | controller | - ' fi' 2026-03-09 19:10:03.692624 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.692628 | controller | - ' ' 2026-03-09 19:10:03.692632 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.692639 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.692644 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.692648 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.692652 | controller | --timeout=300s' 2026-03-09 19:10:03.692657 | controller | - ' ' 2026-03-09 19:10:03.692661 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.692665 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.692670 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.692674 | controller | - ' ' 2026-03-09 19:10:03.692680 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692684 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692688 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.692692 | controller | - ' ' 2026-03-09 19:10:03.692697 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.692701 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.692705 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.692709 | controller | available"' 2026-03-09 19:10:03.692714 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.692718 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.692722 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.692727 | controller | - ' exit 1' 2026-03-09 19:10:03.692731 | controller | - ' fi' 2026-03-09 19:10:03.692735 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.692739 | controller | - ' ' 2026-03-09 19:10:03.692743 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.692748 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.692752 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692757 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.692763 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692769 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.692783 | controller | -- \' 2026-03-09 19:10:03.692790 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.692796 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.692802 | controller | - ' logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.692808 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.692814 | controller | - ' logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.692820 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.692825 | controller | found' 2026-03-09 19:10:03.692831 | controller | - ' logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.692836 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.692845 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.692850 | controller | - ' ' 2026-03-09 19:10:03.692856 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.692861 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.692867 | controller | - ' ' 2026-03-09 19:10:03.692873 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.692878 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.692884 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.692890 | controller | - ' exit 1' 2026-03-09 19:10:03.692895 | controller | - ' fi' 2026-03-09 19:10:03.692900 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.692905 | controller | - ' ' 2026-03-09 19:10:03.692911 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.692916 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.692922 | controller | | base64 -d)' 2026-03-09 19:10:03.692928 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.692934 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.692940 | controller | - ' exit 1' 2026-03-09 19:10:03.692946 | controller | - ' fi' 2026-03-09 19:10:03.692952 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.692958 | controller | - ' ' 2026-03-09 19:10:03.692963 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.692968 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.692974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.692979 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.692985 | controller | --timeout=300s' 2026-03-09 19:10:03.692990 | controller | - ' ' 2026-03-09 19:10:03.692996 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.693001 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.693007 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.693013 | controller | - ' ' 2026-03-09 19:10:03.693019 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693024 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693030 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693036 | controller | - ' ' 2026-03-09 19:10:03.693041 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.693047 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.693053 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.693058 | controller | available"' 2026-03-09 19:10:03.693064 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.693069 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.693075 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.693081 | controller | - ' exit 1' 2026-03-09 19:10:03.693086 | controller | - ' fi' 2026-03-09 19:10:03.693092 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.693098 | controller | - ' ' 2026-03-09 19:10:03.693110 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.693116 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.693123 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693130 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.693136 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693140 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.693166 | controller | -- \' 2026-03-09 19:10:03.693181 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693186 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.693190 | controller | - ' logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.693194 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.693199 | controller | - ' logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.693203 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.693207 | controller | found' 2026-03-09 19:10:03.693211 | controller | - ' logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.693216 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.693220 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.693224 | controller | - ' ' 2026-03-09 19:10:03.693229 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.693233 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.693237 | controller | - ' ' 2026-03-09 19:10:03.693241 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.693246 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.693250 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.693254 | controller | - ' exit 1' 2026-03-09 19:10:03.693259 | controller | - ' fi' 2026-03-09 19:10:03.693263 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.693269 | controller | - ' ' 2026-03-09 19:10:03.693274 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.693278 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.693282 | controller | | base64 -d)' 2026-03-09 19:10:03.693287 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.693291 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.693295 | controller | - ' exit 1' 2026-03-09 19:10:03.693299 | controller | - ' fi' 2026-03-09 19:10:03.693304 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.693308 | controller | - ' ' 2026-03-09 19:10:03.693312 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.693316 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.693321 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.693325 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.693329 | controller | --timeout=300s' 2026-03-09 19:10:03.693333 | controller | - ' ' 2026-03-09 19:10:03.693338 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.693342 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.693348 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.693353 | controller | - ' ' 2026-03-09 19:10:03.693357 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693361 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693366 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693370 | controller | - ' ' 2026-03-09 19:10:03.693374 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.693378 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.693383 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.693387 | controller | available"' 2026-03-09 19:10:03.693391 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.693396 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.693400 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.693404 | controller | - ' exit 1' 2026-03-09 19:10:03.693408 | controller | - ' fi' 2026-03-09 19:10:03.693413 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.693417 | controller | - ' ' 2026-03-09 19:10:03.693421 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.693425 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.693430 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693434 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.693438 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693443 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.693447 | controller | -- \' 2026-03-09 19:10:03.693461 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693466 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.693470 | controller | - ' logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.693475 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.693479 | controller | - ' logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.693485 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.693490 | controller | found' 2026-03-09 19:10:03.693494 | controller | - ' logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.693499 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.693503 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.693507 | controller | - ' ' 2026-03-09 19:10:03.693514 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.693518 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.693522 | controller | - ' ' 2026-03-09 19:10:03.693527 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.693531 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.693535 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.693542 | controller | - ' exit 1' 2026-03-09 19:10:03.693546 | controller | - ' fi' 2026-03-09 19:10:03.693550 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.693554 | controller | - ' ' 2026-03-09 19:10:03.693559 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.693563 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.693567 | controller | | base64 -d)' 2026-03-09 19:10:03.693572 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.693576 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.693580 | controller | - ' exit 1' 2026-03-09 19:10:03.693584 | controller | - ' fi' 2026-03-09 19:10:03.693589 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.693593 | controller | - ' ' 2026-03-09 19:10:03.693597 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.693602 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.693606 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.693610 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.693614 | controller | --timeout=300s' 2026-03-09 19:10:03.693619 | controller | - ' ' 2026-03-09 19:10:03.693623 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.693627 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.693631 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.693636 | controller | - ' ' 2026-03-09 19:10:03.693640 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693644 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693649 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693653 | controller | - ' ' 2026-03-09 19:10:03.693657 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.693661 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.693666 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.693670 | controller | available"' 2026-03-09 19:10:03.693674 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.693678 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.693684 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.693690 | controller | - ' exit 1' 2026-03-09 19:10:03.693694 | controller | - ' fi' 2026-03-09 19:10:03.693698 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.693702 | controller | - ' ' 2026-03-09 19:10:03.693707 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.693711 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.693715 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693720 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.693724 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693730 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.693734 | controller | -- \' 2026-03-09 19:10:03.693738 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.693750 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.693754 | controller | - ' logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.693759 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.693763 | controller | - ' logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.693767 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.693772 | controller | found' 2026-03-09 19:10:03.693776 | controller | - ' logger.go:42: 18:52:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.693780 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.693784 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.693789 | controller | - ' ' 2026-03-09 19:10:03.693793 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.693797 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.693802 | controller | - ' ' 2026-03-09 19:10:03.693806 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.693810 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.693815 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.693819 | controller | - ' exit 1' 2026-03-09 19:10:03.693823 | controller | - ' fi' 2026-03-09 19:10:03.693827 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.693832 | controller | - ' ' 2026-03-09 19:10:03.693839 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.693844 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.693848 | controller | | base64 -d)' 2026-03-09 19:10:03.693852 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.693857 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.693861 | controller | - ' exit 1' 2026-03-09 19:10:03.693865 | controller | - ' fi' 2026-03-09 19:10:03.693869 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.693874 | controller | - ' ' 2026-03-09 19:10:03.693878 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.693882 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.693888 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.693892 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.693897 | controller | --timeout=300s' 2026-03-09 19:10:03.693901 | controller | - ' ' 2026-03-09 19:10:03.693905 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.693909 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.693914 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.693918 | controller | - ' ' 2026-03-09 19:10:03.693922 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693926 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693933 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.693937 | controller | - ' ' 2026-03-09 19:10:03.693941 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.693946 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.693951 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.693957 | controller | available"' 2026-03-09 19:10:03.693961 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.693965 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.693970 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.693974 | controller | - ' exit 1' 2026-03-09 19:10:03.693978 | controller | - ' fi' 2026-03-09 19:10:03.693982 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.693987 | controller | - ' ' 2026-03-09 19:10:03.693991 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.693995 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.693999 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694004 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.694008 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694013 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.694018 | controller | -- \' 2026-03-09 19:10:03.694023 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694034 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.694040 | controller | - ' logger.go:42: 18:52:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.694046 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.694050 | controller | - ' logger.go:42: 18:52:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.694054 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.694059 | controller | found' 2026-03-09 19:10:03.694063 | controller | - ' logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.694067 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.694071 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.694076 | controller | - ' ' 2026-03-09 19:10:03.694080 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.694084 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.694088 | controller | - ' ' 2026-03-09 19:10:03.694093 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.694097 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.694101 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.694106 | controller | - ' exit 1' 2026-03-09 19:10:03.694110 | controller | - ' fi' 2026-03-09 19:10:03.694114 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.694118 | controller | - ' ' 2026-03-09 19:10:03.694123 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.694129 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.694134 | controller | | base64 -d)' 2026-03-09 19:10:03.694138 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.694157 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.694164 | controller | - ' exit 1' 2026-03-09 19:10:03.694168 | controller | - ' fi' 2026-03-09 19:10:03.694173 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.694177 | controller | - ' ' 2026-03-09 19:10:03.694181 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.694186 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.694190 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.694194 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.694198 | controller | --timeout=300s' 2026-03-09 19:10:03.694202 | controller | - ' ' 2026-03-09 19:10:03.694207 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.694211 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.694215 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.694220 | controller | - ' ' 2026-03-09 19:10:03.694224 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694228 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694236 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694241 | controller | - ' ' 2026-03-09 19:10:03.694245 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.694249 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.694254 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.694258 | controller | available"' 2026-03-09 19:10:03.694262 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.694266 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.694271 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.694275 | controller | - ' exit 1' 2026-03-09 19:10:03.694279 | controller | - ' fi' 2026-03-09 19:10:03.694283 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.694288 | controller | - ' ' 2026-03-09 19:10:03.694292 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.694296 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.694300 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694305 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.694309 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694313 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.694317 | controller | -- \' 2026-03-09 19:10:03.694322 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694326 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.694338 | controller | - ' logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.694344 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.694349 | controller | - ' logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.694353 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.694357 | controller | found' 2026-03-09 19:10:03.694362 | controller | - ' logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.694366 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.694370 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.694374 | controller | - ' ' 2026-03-09 19:10:03.694379 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.694383 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.694387 | controller | - ' ' 2026-03-09 19:10:03.694392 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.694396 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.694400 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.694404 | controller | - ' exit 1' 2026-03-09 19:10:03.694409 | controller | - ' fi' 2026-03-09 19:10:03.694413 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.694417 | controller | - ' ' 2026-03-09 19:10:03.694422 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.694426 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.694430 | controller | | base64 -d)' 2026-03-09 19:10:03.694434 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.694439 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.694443 | controller | - ' exit 1' 2026-03-09 19:10:03.694447 | controller | - ' fi' 2026-03-09 19:10:03.694451 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.694456 | controller | - ' ' 2026-03-09 19:10:03.694460 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.694464 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.694469 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.694475 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.694479 | controller | --timeout=300s' 2026-03-09 19:10:03.694483 | controller | - ' ' 2026-03-09 19:10:03.694487 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.694492 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.694496 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.694500 | controller | - ' ' 2026-03-09 19:10:03.694504 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694509 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694513 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694517 | controller | - ' ' 2026-03-09 19:10:03.694524 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.694528 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.694532 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.694537 | controller | available"' 2026-03-09 19:10:03.694541 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.694547 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.694551 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.694555 | controller | - ' exit 1' 2026-03-09 19:10:03.694560 | controller | - ' fi' 2026-03-09 19:10:03.694564 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.694568 | controller | - ' ' 2026-03-09 19:10:03.694573 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.694577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.694581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694585 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.694590 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694594 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.694598 | controller | -- \' 2026-03-09 19:10:03.694602 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694607 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.694618 | controller | - ' logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.694622 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.694626 | controller | - ' logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.694631 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.694635 | controller | found' 2026-03-09 19:10:03.694640 | controller | - ' logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.694644 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.694648 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.694652 | controller | - ' ' 2026-03-09 19:10:03.694657 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.694661 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.694665 | controller | - ' ' 2026-03-09 19:10:03.694669 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.694673 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.694678 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.694682 | controller | - ' exit 1' 2026-03-09 19:10:03.694686 | controller | - ' fi' 2026-03-09 19:10:03.694690 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.694695 | controller | - ' ' 2026-03-09 19:10:03.694699 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.694703 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.694708 | controller | | base64 -d)' 2026-03-09 19:10:03.694712 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.694716 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.694720 | controller | - ' exit 1' 2026-03-09 19:10:03.694725 | controller | - ' fi' 2026-03-09 19:10:03.694729 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.694733 | controller | - ' ' 2026-03-09 19:10:03.694737 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.694742 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.694748 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.694771 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.694775 | controller | --timeout=300s' 2026-03-09 19:10:03.694780 | controller | - ' ' 2026-03-09 19:10:03.694784 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.694788 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.694792 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.694797 | controller | - ' ' 2026-03-09 19:10:03.694801 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694805 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694809 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.694813 | controller | - ' ' 2026-03-09 19:10:03.694818 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.694822 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.694826 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.694830 | controller | available"' 2026-03-09 19:10:03.694835 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.694839 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.694843 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.694847 | controller | - ' exit 1' 2026-03-09 19:10:03.694852 | controller | - ' fi' 2026-03-09 19:10:03.694856 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.694860 | controller | - ' ' 2026-03-09 19:10:03.694864 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.694869 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.694873 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694877 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.694881 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694885 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.694890 | controller | -- \' 2026-03-09 19:10:03.694894 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.694898 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.694903 | controller | - ' logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.694907 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.694917 | controller | - ' logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.694922 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.694926 | controller | found' 2026-03-09 19:10:03.694931 | controller | - ' logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.694935 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.694939 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.694944 | controller | - ' ' 2026-03-09 19:10:03.694950 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.694954 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.694958 | controller | - ' ' 2026-03-09 19:10:03.694963 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.694967 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.694971 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.694975 | controller | - ' exit 1' 2026-03-09 19:10:03.694980 | controller | - ' fi' 2026-03-09 19:10:03.694984 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.694988 | controller | - ' ' 2026-03-09 19:10:03.694992 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.694997 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.695001 | controller | | base64 -d)' 2026-03-09 19:10:03.695005 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.695010 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.695014 | controller | - ' exit 1' 2026-03-09 19:10:03.695018 | controller | - ' fi' 2026-03-09 19:10:03.695022 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.695027 | controller | - ' ' 2026-03-09 19:10:03.695031 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.695035 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.695039 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.695044 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.695048 | controller | --timeout=300s' 2026-03-09 19:10:03.695052 | controller | - ' ' 2026-03-09 19:10:03.695056 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.695061 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.695065 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.695069 | controller | - ' ' 2026-03-09 19:10:03.695073 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695078 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695082 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695086 | controller | - ' ' 2026-03-09 19:10:03.695092 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.695097 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.695101 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.695105 | controller | available"' 2026-03-09 19:10:03.695109 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.695115 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.695119 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.695123 | controller | - ' exit 1' 2026-03-09 19:10:03.695128 | controller | - ' fi' 2026-03-09 19:10:03.695132 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.695136 | controller | - ' ' 2026-03-09 19:10:03.695140 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.695158 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.695164 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695169 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.695173 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695177 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.695181 | controller | -- \' 2026-03-09 19:10:03.695186 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695190 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.695196 | controller | - ' logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.695200 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.695212 | controller | - ' logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.695216 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.695220 | controller | found' 2026-03-09 19:10:03.695225 | controller | - ' logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.695229 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.695234 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.695238 | controller | - ' ' 2026-03-09 19:10:03.695242 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.695246 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.695251 | controller | - ' ' 2026-03-09 19:10:03.695255 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.695259 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.695263 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.695268 | controller | - ' exit 1' 2026-03-09 19:10:03.695272 | controller | - ' fi' 2026-03-09 19:10:03.695276 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.695280 | controller | - ' ' 2026-03-09 19:10:03.695285 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.695289 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.695293 | controller | | base64 -d)' 2026-03-09 19:10:03.695297 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.695302 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.695306 | controller | - ' exit 1' 2026-03-09 19:10:03.695310 | controller | - ' fi' 2026-03-09 19:10:03.695314 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.695319 | controller | - ' ' 2026-03-09 19:10:03.695323 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.695327 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.695331 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.695336 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.695340 | controller | --timeout=300s' 2026-03-09 19:10:03.695344 | controller | - ' ' 2026-03-09 19:10:03.695348 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.695353 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.695360 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.695364 | controller | - ' ' 2026-03-09 19:10:03.695368 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695373 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695377 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695381 | controller | - ' ' 2026-03-09 19:10:03.695385 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.695390 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.695394 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.695398 | controller | available"' 2026-03-09 19:10:03.695403 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.695407 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.695411 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.695415 | controller | - ' exit 1' 2026-03-09 19:10:03.695420 | controller | - ' fi' 2026-03-09 19:10:03.695424 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.695428 | controller | - ' ' 2026-03-09 19:10:03.695432 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.695437 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.695441 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695445 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.695449 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695454 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.695458 | controller | -- \' 2026-03-09 19:10:03.695462 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695466 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.695472 | controller | - ' logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.695477 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.695481 | controller | - ' logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.695492 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.695496 | controller | found' 2026-03-09 19:10:03.695500 | controller | - ' logger.go:42: 18:52:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.695505 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.695509 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.695515 | controller | - ' ' 2026-03-09 19:10:03.695520 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.695524 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.695528 | controller | - ' ' 2026-03-09 19:10:03.695533 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.695538 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.695542 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.695552 | controller | - ' exit 1' 2026-03-09 19:10:03.695557 | controller | - ' fi' 2026-03-09 19:10:03.695561 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.695565 | controller | - ' ' 2026-03-09 19:10:03.695570 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.695574 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.695578 | controller | | base64 -d)' 2026-03-09 19:10:03.695583 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.695587 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.695591 | controller | - ' exit 1' 2026-03-09 19:10:03.695596 | controller | - ' fi' 2026-03-09 19:10:03.695602 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.695606 | controller | - ' ' 2026-03-09 19:10:03.695610 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.695615 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.695619 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.695623 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.695628 | controller | --timeout=300s' 2026-03-09 19:10:03.695632 | controller | - ' ' 2026-03-09 19:10:03.695636 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.695641 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.695645 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.695649 | controller | - ' ' 2026-03-09 19:10:03.695654 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695658 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695662 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695666 | controller | - ' ' 2026-03-09 19:10:03.695671 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.695675 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.695681 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.695685 | controller | available"' 2026-03-09 19:10:03.695690 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.695694 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.695698 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.695702 | controller | - ' exit 1' 2026-03-09 19:10:03.695707 | controller | - ' fi' 2026-03-09 19:10:03.695711 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.695715 | controller | - ' ' 2026-03-09 19:10:03.695720 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.695724 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.695749 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695756 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.695760 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695764 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.695771 | controller | -- \' 2026-03-09 19:10:03.695775 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.695779 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.695783 | controller | - ' logger.go:42: 18:52:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.695788 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.695794 | controller | - ' logger.go:42: 18:52:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.695806 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.695810 | controller | found' 2026-03-09 19:10:03.695815 | controller | - ' logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.695819 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.695824 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.695828 | controller | - ' ' 2026-03-09 19:10:03.695832 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.695836 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.695841 | controller | - ' ' 2026-03-09 19:10:03.695846 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.695852 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.695857 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.695861 | controller | - ' exit 1' 2026-03-09 19:10:03.695865 | controller | - ' fi' 2026-03-09 19:10:03.695870 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.695874 | controller | - ' ' 2026-03-09 19:10:03.695878 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.695882 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.695887 | controller | | base64 -d)' 2026-03-09 19:10:03.695891 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.695895 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.695899 | controller | - ' exit 1' 2026-03-09 19:10:03.695904 | controller | - ' fi' 2026-03-09 19:10:03.695908 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.695912 | controller | - ' ' 2026-03-09 19:10:03.695916 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.695921 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.695925 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.695929 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.695933 | controller | --timeout=300s' 2026-03-09 19:10:03.695938 | controller | - ' ' 2026-03-09 19:10:03.695942 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.695946 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.695950 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.695955 | controller | - ' ' 2026-03-09 19:10:03.695959 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695963 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695967 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.695974 | controller | - ' ' 2026-03-09 19:10:03.695978 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.695982 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.695987 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.695991 | controller | available"' 2026-03-09 19:10:03.695995 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.696000 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.696004 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.696008 | controller | - ' exit 1' 2026-03-09 19:10:03.696012 | controller | - ' fi' 2026-03-09 19:10:03.696017 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.696021 | controller | - ' ' 2026-03-09 19:10:03.696025 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.696029 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.696034 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696038 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.696042 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696046 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.696051 | controller | -- \' 2026-03-09 19:10:03.696055 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696059 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.696063 | controller | - ' logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.696068 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.696072 | controller | - ' logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.696076 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.696080 | controller | found' 2026-03-09 19:10:03.696091 | controller | - ' logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.696096 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.696100 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.696106 | controller | - ' ' 2026-03-09 19:10:03.696111 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.696116 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.696120 | controller | - ' ' 2026-03-09 19:10:03.696126 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.696131 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.696135 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.696140 | controller | - ' exit 1' 2026-03-09 19:10:03.696162 | controller | - ' fi' 2026-03-09 19:10:03.696169 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.696174 | controller | - ' ' 2026-03-09 19:10:03.696178 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.696182 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.696187 | controller | | base64 -d)' 2026-03-09 19:10:03.696191 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.696197 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.696202 | controller | - ' exit 1' 2026-03-09 19:10:03.696206 | controller | - ' fi' 2026-03-09 19:10:03.696210 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.696215 | controller | - ' ' 2026-03-09 19:10:03.696219 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.696223 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.696227 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.696232 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.696236 | controller | --timeout=300s' 2026-03-09 19:10:03.696240 | controller | - ' ' 2026-03-09 19:10:03.696244 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.696249 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.696253 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.696257 | controller | - ' ' 2026-03-09 19:10:03.696262 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696266 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696270 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696274 | controller | - ' ' 2026-03-09 19:10:03.696279 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.696285 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.696289 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.696294 | controller | available"' 2026-03-09 19:10:03.696298 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.696302 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.696307 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.696311 | controller | - ' exit 1' 2026-03-09 19:10:03.696315 | controller | - ' fi' 2026-03-09 19:10:03.696320 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.696325 | controller | - ' ' 2026-03-09 19:10:03.696330 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.696334 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.696338 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696343 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.696347 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696351 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.696355 | controller | -- \' 2026-03-09 19:10:03.696360 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696364 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.696368 | controller | - ' logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.696372 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.696378 | controller | - ' logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.696383 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.696387 | controller | found' 2026-03-09 19:10:03.696392 | controller | - ' logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.696408 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.696415 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.696421 | controller | - ' ' 2026-03-09 19:10:03.696426 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.696431 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.696435 | controller | - ' ' 2026-03-09 19:10:03.696439 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.696443 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.696448 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.696452 | controller | - ' exit 1' 2026-03-09 19:10:03.696456 | controller | - ' fi' 2026-03-09 19:10:03.696460 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.696465 | controller | - ' ' 2026-03-09 19:10:03.696471 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.696476 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.696480 | controller | | base64 -d)' 2026-03-09 19:10:03.696484 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.696488 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.696493 | controller | - ' exit 1' 2026-03-09 19:10:03.696497 | controller | - ' fi' 2026-03-09 19:10:03.696501 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.696506 | controller | - ' ' 2026-03-09 19:10:03.696510 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.696514 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.696518 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.696523 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.696527 | controller | --timeout=300s' 2026-03-09 19:10:03.696531 | controller | - ' ' 2026-03-09 19:10:03.696535 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.696540 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.696544 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.696548 | controller | - ' ' 2026-03-09 19:10:03.696553 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696557 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696561 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696565 | controller | - ' ' 2026-03-09 19:10:03.696570 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.696574 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.696578 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.696582 | controller | available"' 2026-03-09 19:10:03.696587 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.696593 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.696597 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.696601 | controller | - ' exit 1' 2026-03-09 19:10:03.696606 | controller | - ' fi' 2026-03-09 19:10:03.696610 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.696614 | controller | - ' ' 2026-03-09 19:10:03.696618 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.696623 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.696627 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696631 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.696635 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696640 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.696646 | controller | -- \' 2026-03-09 19:10:03.696650 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696654 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.696658 | controller | - ' logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.696663 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.696667 | controller | - ' logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.696671 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.696676 | controller | found' 2026-03-09 19:10:03.696680 | controller | - ' logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.696684 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.696696 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.696701 | controller | - ' ' 2026-03-09 19:10:03.696705 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.696710 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.696714 | controller | - ' ' 2026-03-09 19:10:03.696718 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.696723 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.696727 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.696731 | controller | - ' exit 1' 2026-03-09 19:10:03.696737 | controller | - ' fi' 2026-03-09 19:10:03.696742 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.696746 | controller | - ' ' 2026-03-09 19:10:03.696750 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.696755 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.696759 | controller | | base64 -d)' 2026-03-09 19:10:03.696763 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.696768 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.696772 | controller | - ' exit 1' 2026-03-09 19:10:03.696776 | controller | - ' fi' 2026-03-09 19:10:03.696780 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.696785 | controller | - ' ' 2026-03-09 19:10:03.696789 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.696793 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.696798 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.696804 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.696808 | controller | --timeout=300s' 2026-03-09 19:10:03.696812 | controller | - ' ' 2026-03-09 19:10:03.696817 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.696821 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.696825 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.696829 | controller | - ' ' 2026-03-09 19:10:03.696837 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696841 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696845 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.696850 | controller | - ' ' 2026-03-09 19:10:03.696854 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.696858 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.696862 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.696867 | controller | available"' 2026-03-09 19:10:03.696871 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.696875 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.696880 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.696884 | controller | - ' exit 1' 2026-03-09 19:10:03.696888 | controller | - ' fi' 2026-03-09 19:10:03.696892 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.696897 | controller | - ' ' 2026-03-09 19:10:03.696903 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.696907 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.696912 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696916 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.696920 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696924 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.696929 | controller | -- \' 2026-03-09 19:10:03.696933 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.696937 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.696942 | controller | - ' logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.696946 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.696950 | controller | - ' logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.696955 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.696959 | controller | found' 2026-03-09 19:10:03.696963 | controller | - ' logger.go:42: 18:52:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.696968 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.696972 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.696976 | controller | - ' ' 2026-03-09 19:10:03.696987 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.696994 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.696998 | controller | - ' ' 2026-03-09 19:10:03.697002 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.697007 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.697011 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.697016 | controller | - ' exit 1' 2026-03-09 19:10:03.697020 | controller | - ' fi' 2026-03-09 19:10:03.697024 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.697029 | controller | - ' ' 2026-03-09 19:10:03.697035 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.697041 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.697047 | controller | | base64 -d)' 2026-03-09 19:10:03.697053 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.697059 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.697063 | controller | - ' exit 1' 2026-03-09 19:10:03.697068 | controller | - ' fi' 2026-03-09 19:10:03.697072 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.697076 | controller | - ' ' 2026-03-09 19:10:03.697080 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.697085 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.697089 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.697093 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.697097 | controller | --timeout=300s' 2026-03-09 19:10:03.697102 | controller | - ' ' 2026-03-09 19:10:03.697106 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.697110 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.697114 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.697119 | controller | - ' ' 2026-03-09 19:10:03.697123 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697127 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697131 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697136 | controller | - ' ' 2026-03-09 19:10:03.697140 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.697156 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.697163 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.697168 | controller | available"' 2026-03-09 19:10:03.697172 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.697176 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.697180 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.697185 | controller | - ' exit 1' 2026-03-09 19:10:03.697189 | controller | - ' fi' 2026-03-09 19:10:03.697193 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.697197 | controller | - ' ' 2026-03-09 19:10:03.697202 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.697206 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.697213 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697217 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.697221 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697225 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.697230 | controller | -- \' 2026-03-09 19:10:03.697234 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697238 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.697242 | controller | - ' logger.go:42: 18:52:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.697247 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.697251 | controller | - ' logger.go:42: 18:52:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.697255 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.697259 | controller | found' 2026-03-09 19:10:03.697264 | controller | - ' logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.697268 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.697272 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.697277 | controller | - ' ' 2026-03-09 19:10:03.697289 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.697294 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.697298 | controller | - ' ' 2026-03-09 19:10:03.697302 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.697307 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.697311 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.697315 | controller | - ' exit 1' 2026-03-09 19:10:03.697319 | controller | - ' fi' 2026-03-09 19:10:03.697324 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.697328 | controller | - ' ' 2026-03-09 19:10:03.697332 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.697337 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.697341 | controller | | base64 -d)' 2026-03-09 19:10:03.697345 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.697349 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.697354 | controller | - ' exit 1' 2026-03-09 19:10:03.697358 | controller | - ' fi' 2026-03-09 19:10:03.697362 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.697367 | controller | - ' ' 2026-03-09 19:10:03.697371 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.697375 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.697379 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.697384 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.697388 | controller | --timeout=300s' 2026-03-09 19:10:03.697392 | controller | - ' ' 2026-03-09 19:10:03.697396 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.697401 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.697409 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.697413 | controller | - ' ' 2026-03-09 19:10:03.697418 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697422 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697426 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697430 | controller | - ' ' 2026-03-09 19:10:03.697435 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.697439 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.697443 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.697447 | controller | available"' 2026-03-09 19:10:03.697451 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.697456 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.697460 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.697464 | controller | - ' exit 1' 2026-03-09 19:10:03.697468 | controller | - ' fi' 2026-03-09 19:10:03.697473 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.697477 | controller | - ' ' 2026-03-09 19:10:03.697481 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.697485 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.697490 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697496 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.697500 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697504 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.697508 | controller | -- \' 2026-03-09 19:10:03.697513 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697517 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.697521 | controller | - ' logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.697526 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.697530 | controller | - ' logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.697534 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.697538 | controller | found' 2026-03-09 19:10:03.697543 | controller | - ' logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.697547 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.697551 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.697555 | controller | - ' ' 2026-03-09 19:10:03.697560 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.697573 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.697578 | controller | - ' ' 2026-03-09 19:10:03.697582 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.697586 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.697591 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.697595 | controller | - ' exit 1' 2026-03-09 19:10:03.697601 | controller | - ' fi' 2026-03-09 19:10:03.697605 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.697610 | controller | - ' ' 2026-03-09 19:10:03.697614 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.697618 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.697622 | controller | | base64 -d)' 2026-03-09 19:10:03.697627 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.697631 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.697635 | controller | - ' exit 1' 2026-03-09 19:10:03.697640 | controller | - ' fi' 2026-03-09 19:10:03.697644 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.697648 | controller | - ' ' 2026-03-09 19:10:03.697653 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.697657 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.697661 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.697665 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.697670 | controller | --timeout=300s' 2026-03-09 19:10:03.697674 | controller | - ' ' 2026-03-09 19:10:03.697678 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.697682 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.697687 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.697691 | controller | - ' ' 2026-03-09 19:10:03.697695 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697699 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697704 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697708 | controller | - ' ' 2026-03-09 19:10:03.697712 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.697717 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.697721 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.697725 | controller | available"' 2026-03-09 19:10:03.697729 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.697736 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.697741 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.697745 | controller | - ' exit 1' 2026-03-09 19:10:03.697751 | controller | - ' fi' 2026-03-09 19:10:03.697755 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.697759 | controller | - ' ' 2026-03-09 19:10:03.697764 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.697768 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.697772 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697777 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.697781 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697785 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.697791 | controller | -- \' 2026-03-09 19:10:03.697795 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.697800 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.697804 | controller | - ' logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.697808 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.697812 | controller | - ' logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.697817 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.697821 | controller | found' 2026-03-09 19:10:03.697825 | controller | - ' logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.697829 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.697834 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.697838 | controller | - ' ' 2026-03-09 19:10:03.697842 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.697847 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.697858 | controller | - ' ' 2026-03-09 19:10:03.697862 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.697866 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.697871 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.697875 | controller | - ' exit 1' 2026-03-09 19:10:03.697879 | controller | - ' fi' 2026-03-09 19:10:03.697884 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.697888 | controller | - ' ' 2026-03-09 19:10:03.697892 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.697897 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.697901 | controller | | base64 -d)' 2026-03-09 19:10:03.697905 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.697910 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.697914 | controller | - ' exit 1' 2026-03-09 19:10:03.697918 | controller | - ' fi' 2026-03-09 19:10:03.697922 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.697927 | controller | - ' ' 2026-03-09 19:10:03.697931 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.697935 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.697939 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.697944 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.697949 | controller | --timeout=300s' 2026-03-09 19:10:03.697955 | controller | - ' ' 2026-03-09 19:10:03.697960 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.697964 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.697968 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.697972 | controller | - ' ' 2026-03-09 19:10:03.697977 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697981 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697985 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.697992 | controller | - ' ' 2026-03-09 19:10:03.697996 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.698000 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.698005 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.698009 | controller | available"' 2026-03-09 19:10:03.698013 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.698018 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.698022 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.698026 | controller | - ' exit 1' 2026-03-09 19:10:03.698030 | controller | - ' fi' 2026-03-09 19:10:03.698035 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.698039 | controller | - ' ' 2026-03-09 19:10:03.698043 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.698048 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.698052 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698056 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.698060 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698065 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.698069 | controller | -- \' 2026-03-09 19:10:03.698073 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698079 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.698083 | controller | - ' logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.698088 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.698092 | controller | - ' logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.698096 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.698101 | controller | found' 2026-03-09 19:10:03.698105 | controller | - ' logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.698109 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.698113 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.698118 | controller | - ' ' 2026-03-09 19:10:03.698122 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.698126 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.698130 | controller | - ' ' 2026-03-09 19:10:03.698141 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.698157 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.698163 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.698167 | controller | - ' exit 1' 2026-03-09 19:10:03.698172 | controller | - ' fi' 2026-03-09 19:10:03.698176 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.698180 | controller | - ' ' 2026-03-09 19:10:03.698185 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.698189 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.698193 | controller | | base64 -d)' 2026-03-09 19:10:03.698197 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.698204 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.698210 | controller | - ' exit 1' 2026-03-09 19:10:03.698214 | controller | - ' fi' 2026-03-09 19:10:03.698219 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.698223 | controller | - ' ' 2026-03-09 19:10:03.698227 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.698231 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.698236 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.698240 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.698244 | controller | --timeout=300s' 2026-03-09 19:10:03.698249 | controller | - ' ' 2026-03-09 19:10:03.698253 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.698257 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.698262 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.698266 | controller | - ' ' 2026-03-09 19:10:03.698270 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698275 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698279 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698283 | controller | - ' ' 2026-03-09 19:10:03.698287 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.698292 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.698296 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.698301 | controller | available"' 2026-03-09 19:10:03.698305 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.698309 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.698313 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.698318 | controller | - ' exit 1' 2026-03-09 19:10:03.698322 | controller | - ' fi' 2026-03-09 19:10:03.698327 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.698331 | controller | - ' ' 2026-03-09 19:10:03.698335 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.698340 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.698344 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698348 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.698355 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698360 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.698364 | controller | -- \' 2026-03-09 19:10:03.698368 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698373 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.698377 | controller | - ' logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.698381 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.698386 | controller | - ' logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.698392 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.698397 | controller | found' 2026-03-09 19:10:03.698401 | controller | - ' logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.698406 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.698410 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.698414 | controller | - ' ' 2026-03-09 19:10:03.698419 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.698423 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.698427 | controller | - ' ' 2026-03-09 19:10:03.698439 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.698444 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.698448 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.698452 | controller | - ' exit 1' 2026-03-09 19:10:03.698457 | controller | - ' fi' 2026-03-09 19:10:03.698461 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.698465 | controller | - ' ' 2026-03-09 19:10:03.698470 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.698474 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.698479 | controller | | base64 -d)' 2026-03-09 19:10:03.698483 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.698487 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.698491 | controller | - ' exit 1' 2026-03-09 19:10:03.698496 | controller | - ' fi' 2026-03-09 19:10:03.698500 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.698505 | controller | - ' ' 2026-03-09 19:10:03.698509 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.698513 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.698517 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.698522 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.698526 | controller | --timeout=300s' 2026-03-09 19:10:03.698530 | controller | - ' ' 2026-03-09 19:10:03.698535 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.698539 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.698543 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.698548 | controller | - ' ' 2026-03-09 19:10:03.698552 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698556 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698561 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698565 | controller | - ' ' 2026-03-09 19:10:03.698569 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.698574 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.698578 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.698582 | controller | available"' 2026-03-09 19:10:03.698587 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.698591 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.698597 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.698601 | controller | - ' exit 1' 2026-03-09 19:10:03.698606 | controller | - ' fi' 2026-03-09 19:10:03.698610 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.698614 | controller | - ' ' 2026-03-09 19:10:03.698619 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.698623 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.698627 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698632 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.698636 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698640 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.698644 | controller | -- \' 2026-03-09 19:10:03.698649 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698653 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.698657 | controller | - ' logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.698662 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.698668 | controller | - ' logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.698673 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.698678 | controller | found' 2026-03-09 19:10:03.698682 | controller | - ' logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.698686 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.698691 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.698695 | controller | - ' ' 2026-03-09 19:10:03.698699 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.698704 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.698708 | controller | - ' ' 2026-03-09 19:10:03.698712 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.698717 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.698727 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.698732 | controller | - ' exit 1' 2026-03-09 19:10:03.698736 | controller | - ' fi' 2026-03-09 19:10:03.698741 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.698745 | controller | - ' ' 2026-03-09 19:10:03.698749 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.698754 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.698758 | controller | | base64 -d)' 2026-03-09 19:10:03.698763 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.698767 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.698771 | controller | - ' exit 1' 2026-03-09 19:10:03.698776 | controller | - ' fi' 2026-03-09 19:10:03.698780 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.698784 | controller | - ' ' 2026-03-09 19:10:03.698789 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.698793 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.698797 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.698804 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.698809 | controller | --timeout=300s' 2026-03-09 19:10:03.698815 | controller | - ' ' 2026-03-09 19:10:03.698819 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.698823 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.698828 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.698832 | controller | - ' ' 2026-03-09 19:10:03.698836 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698841 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698845 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.698849 | controller | - ' ' 2026-03-09 19:10:03.698854 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.698858 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.698862 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.698866 | controller | available"' 2026-03-09 19:10:03.698871 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.698875 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.698879 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.698884 | controller | - ' exit 1' 2026-03-09 19:10:03.698889 | controller | - ' fi' 2026-03-09 19:10:03.698894 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.698900 | controller | - ' ' 2026-03-09 19:10:03.698904 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.698908 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.698913 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698917 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.698921 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698926 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.698930 | controller | -- \' 2026-03-09 19:10:03.698934 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.698938 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.698943 | controller | - ' logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.698947 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.698951 | controller | - ' logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.698956 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.698960 | controller | found' 2026-03-09 19:10:03.698964 | controller | - ' logger.go:42: 18:52:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.698969 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.698973 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.698977 | controller | - ' ' 2026-03-09 19:10:03.698982 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.698988 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.698992 | controller | - ' ' 2026-03-09 19:10:03.698996 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.699002 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.699014 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.699019 | controller | - ' exit 1' 2026-03-09 19:10:03.699024 | controller | - ' fi' 2026-03-09 19:10:03.699028 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.699033 | controller | - ' ' 2026-03-09 19:10:03.699042 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.699049 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.699055 | controller | | base64 -d)' 2026-03-09 19:10:03.699061 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.699066 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.699071 | controller | - ' exit 1' 2026-03-09 19:10:03.699075 | controller | - ' fi' 2026-03-09 19:10:03.699080 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.699084 | controller | - ' ' 2026-03-09 19:10:03.699088 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.699093 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.699097 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.699101 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.699105 | controller | --timeout=300s' 2026-03-09 19:10:03.699110 | controller | - ' ' 2026-03-09 19:10:03.699114 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.699118 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.699123 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.699127 | controller | - ' ' 2026-03-09 19:10:03.699131 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699136 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699140 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699155 | controller | - ' ' 2026-03-09 19:10:03.699162 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.699167 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.699171 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.699175 | controller | available"' 2026-03-09 19:10:03.699180 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.699184 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.699188 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.699193 | controller | - ' exit 1' 2026-03-09 19:10:03.699197 | controller | - ' fi' 2026-03-09 19:10:03.699201 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.699205 | controller | - ' ' 2026-03-09 19:10:03.699210 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.699214 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.699222 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699226 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.699232 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699237 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.699242 | controller | -- \' 2026-03-09 19:10:03.699246 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699251 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.699255 | controller | - ' logger.go:42: 18:52:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.699259 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.699266 | controller | - ' logger.go:42: 18:52:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.699270 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.699276 | controller | found' 2026-03-09 19:10:03.699281 | controller | - ' logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.699285 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.699290 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.699294 | controller | - ' ' 2026-03-09 19:10:03.699299 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.699305 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.699310 | controller | - ' ' 2026-03-09 19:10:03.699314 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.699319 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.699323 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.699327 | controller | - ' exit 1' 2026-03-09 19:10:03.699340 | controller | - ' fi' 2026-03-09 19:10:03.699345 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.699349 | controller | - ' ' 2026-03-09 19:10:03.699353 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.699358 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.699362 | controller | | base64 -d)' 2026-03-09 19:10:03.699367 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.699371 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.699375 | controller | - ' exit 1' 2026-03-09 19:10:03.699380 | controller | - ' fi' 2026-03-09 19:10:03.699384 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.699388 | controller | - ' ' 2026-03-09 19:10:03.699393 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.699397 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.699401 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.699406 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.699410 | controller | --timeout=300s' 2026-03-09 19:10:03.699414 | controller | - ' ' 2026-03-09 19:10:03.699419 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.699423 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.699429 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.699436 | controller | - ' ' 2026-03-09 19:10:03.699441 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699447 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699451 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699456 | controller | - ' ' 2026-03-09 19:10:03.699460 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.699464 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.699469 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.699473 | controller | available"' 2026-03-09 19:10:03.699477 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.699482 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.699486 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.699490 | controller | - ' exit 1' 2026-03-09 19:10:03.699495 | controller | - ' fi' 2026-03-09 19:10:03.699499 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.699504 | controller | - ' ' 2026-03-09 19:10:03.699508 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.699512 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.699517 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699521 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.699525 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699530 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.699534 | controller | -- \' 2026-03-09 19:10:03.699538 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699543 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.699549 | controller | - ' logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.699553 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.699557 | controller | - ' logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.699562 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.699566 | controller | found' 2026-03-09 19:10:03.699570 | controller | - ' logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.699575 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.699579 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.699583 | controller | - ' ' 2026-03-09 19:10:03.699588 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.699592 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.699596 | controller | - ' ' 2026-03-09 19:10:03.699601 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.699605 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.699610 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.699616 | controller | - ' exit 1' 2026-03-09 19:10:03.699621 | controller | - ' fi' 2026-03-09 19:10:03.699632 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.699640 | controller | - ' ' 2026-03-09 19:10:03.699645 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.699649 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.699653 | controller | | base64 -d)' 2026-03-09 19:10:03.699658 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.699662 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.699666 | controller | - ' exit 1' 2026-03-09 19:10:03.699671 | controller | - ' fi' 2026-03-09 19:10:03.699675 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.699679 | controller | - ' ' 2026-03-09 19:10:03.699684 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.699688 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.699692 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.699697 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.699701 | controller | --timeout=300s' 2026-03-09 19:10:03.699707 | controller | - ' ' 2026-03-09 19:10:03.699712 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.699716 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.699721 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.699737 | controller | - ' ' 2026-03-09 19:10:03.699744 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699750 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699754 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.699758 | controller | - ' ' 2026-03-09 19:10:03.699763 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.699767 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.699771 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.699776 | controller | available"' 2026-03-09 19:10:03.699782 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.699787 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.699791 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.699795 | controller | - ' exit 1' 2026-03-09 19:10:03.699800 | controller | - ' fi' 2026-03-09 19:10:03.699804 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.699808 | controller | - ' ' 2026-03-09 19:10:03.699813 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.699817 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.699821 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699826 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.699830 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699834 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.699838 | controller | -- \' 2026-03-09 19:10:03.699846 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.699851 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.699855 | controller | - ' logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.699859 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.699864 | controller | - ' logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.699868 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.699872 | controller | found' 2026-03-09 19:10:03.699876 | controller | - ' logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.699881 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.699887 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.699891 | controller | - ' ' 2026-03-09 19:10:03.699896 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.699900 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.699904 | controller | - ' ' 2026-03-09 19:10:03.699908 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.699913 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.699917 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.699921 | controller | - ' exit 1' 2026-03-09 19:10:03.699926 | controller | - ' fi' 2026-03-09 19:10:03.699930 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.699934 | controller | - ' ' 2026-03-09 19:10:03.699948 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.699953 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.699957 | controller | | base64 -d)' 2026-03-09 19:10:03.699961 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.699966 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.699970 | controller | - ' exit 1' 2026-03-09 19:10:03.699974 | controller | - ' fi' 2026-03-09 19:10:03.699979 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.699983 | controller | - ' ' 2026-03-09 19:10:03.699987 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.699991 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.699996 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.700000 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.700004 | controller | --timeout=300s' 2026-03-09 19:10:03.700008 | controller | - ' ' 2026-03-09 19:10:03.700013 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.700017 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.700021 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.700025 | controller | - ' ' 2026-03-09 19:10:03.700030 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700034 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700038 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700042 | controller | - ' ' 2026-03-09 19:10:03.700047 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.700053 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.700057 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.700062 | controller | available"' 2026-03-09 19:10:03.700066 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.700070 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.700074 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.700079 | controller | - ' exit 1' 2026-03-09 19:10:03.700083 | controller | - ' fi' 2026-03-09 19:10:03.700087 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.700092 | controller | - ' ' 2026-03-09 19:10:03.700096 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.700100 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.700104 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700109 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.700113 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700117 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.700121 | controller | -- \' 2026-03-09 19:10:03.700126 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700130 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.700134 | controller | - ' logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.700139 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.700143 | controller | - ' logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.700162 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.700166 | controller | found' 2026-03-09 19:10:03.700171 | controller | - ' logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.700175 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.700179 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.700183 | controller | - ' ' 2026-03-09 19:10:03.700188 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.700192 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.700197 | controller | - ' ' 2026-03-09 19:10:03.700203 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.700208 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.700212 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.700216 | controller | - ' exit 1' 2026-03-09 19:10:03.700220 | controller | - ' fi' 2026-03-09 19:10:03.700225 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.700229 | controller | - ' ' 2026-03-09 19:10:03.700233 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.700245 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.700250 | controller | | base64 -d)' 2026-03-09 19:10:03.700254 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.700259 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.700263 | controller | - ' exit 1' 2026-03-09 19:10:03.700269 | controller | - ' fi' 2026-03-09 19:10:03.700274 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.700278 | controller | - ' ' 2026-03-09 19:10:03.700282 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.700286 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.700291 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.700295 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.700299 | controller | --timeout=300s' 2026-03-09 19:10:03.700304 | controller | - ' ' 2026-03-09 19:10:03.700308 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.700312 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.700316 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.700321 | controller | - ' ' 2026-03-09 19:10:03.700325 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700329 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700333 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700338 | controller | - ' ' 2026-03-09 19:10:03.700342 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.700346 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.700350 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.700355 | controller | available"' 2026-03-09 19:10:03.700359 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.700366 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.700370 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.700374 | controller | - ' exit 1' 2026-03-09 19:10:03.700379 | controller | - ' fi' 2026-03-09 19:10:03.700383 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.700387 | controller | - ' ' 2026-03-09 19:10:03.700391 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.700396 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.700400 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700404 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.700408 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700413 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.700419 | controller | -- \' 2026-03-09 19:10:03.700423 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700427 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.700431 | controller | - ' logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.700436 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.700440 | controller | - ' logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.700444 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.700451 | controller | found' 2026-03-09 19:10:03.700455 | controller | - ' logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.700459 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.700464 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.700468 | controller | - ' ' 2026-03-09 19:10:03.700474 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.700478 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.700482 | controller | - ' ' 2026-03-09 19:10:03.700487 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.700491 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.700495 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.700500 | controller | - ' exit 1' 2026-03-09 19:10:03.700504 | controller | - ' fi' 2026-03-09 19:10:03.700508 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.700512 | controller | - ' ' 2026-03-09 19:10:03.700517 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.700528 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.700533 | controller | | base64 -d)' 2026-03-09 19:10:03.700537 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.700541 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.700546 | controller | - ' exit 1' 2026-03-09 19:10:03.700550 | controller | - ' fi' 2026-03-09 19:10:03.700554 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.700558 | controller | - ' ' 2026-03-09 19:10:03.700563 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.700567 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.700571 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.700576 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.700580 | controller | --timeout=300s' 2026-03-09 19:10:03.700584 | controller | - ' ' 2026-03-09 19:10:03.700588 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.700593 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.700597 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.700601 | controller | - ' ' 2026-03-09 19:10:03.700606 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700610 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700614 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700618 | controller | - ' ' 2026-03-09 19:10:03.700623 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.700627 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.700631 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.700635 | controller | available"' 2026-03-09 19:10:03.700640 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.700644 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.700648 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.700654 | controller | - ' exit 1' 2026-03-09 19:10:03.700658 | controller | - ' fi' 2026-03-09 19:10:03.700663 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.700667 | controller | - ' ' 2026-03-09 19:10:03.700671 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.700676 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.700680 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700684 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.700688 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700693 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.700697 | controller | -- \' 2026-03-09 19:10:03.700701 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700706 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.700710 | controller | - ' logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.700714 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.700718 | controller | - ' logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.700723 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.700727 | controller | found' 2026-03-09 19:10:03.700731 | controller | - ' logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.700736 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.700740 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.700744 | controller | - ' ' 2026-03-09 19:10:03.700748 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.700753 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.700757 | controller | - ' ' 2026-03-09 19:10:03.700761 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.700766 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.700770 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.700774 | controller | - ' exit 1' 2026-03-09 19:10:03.700778 | controller | - ' fi' 2026-03-09 19:10:03.700783 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.700787 | controller | - ' ' 2026-03-09 19:10:03.700791 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.700796 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.700807 | controller | | base64 -d)' 2026-03-09 19:10:03.700811 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.700815 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.700820 | controller | - ' exit 1' 2026-03-09 19:10:03.700824 | controller | - ' fi' 2026-03-09 19:10:03.700828 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.700833 | controller | - ' ' 2026-03-09 19:10:03.700837 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.700841 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.700845 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.700850 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.700856 | controller | --timeout=300s' 2026-03-09 19:10:03.700860 | controller | - ' ' 2026-03-09 19:10:03.700865 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.700869 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.700873 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.700878 | controller | - ' ' 2026-03-09 19:10:03.700885 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700891 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700895 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.700899 | controller | - ' ' 2026-03-09 19:10:03.700904 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.700908 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.700912 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.700916 | controller | available"' 2026-03-09 19:10:03.700921 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.700925 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.700929 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.700933 | controller | - ' exit 1' 2026-03-09 19:10:03.700938 | controller | - ' fi' 2026-03-09 19:10:03.700942 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.700946 | controller | - ' ' 2026-03-09 19:10:03.700953 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.700957 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.700961 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700966 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.700970 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700974 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.700979 | controller | -- \' 2026-03-09 19:10:03.700983 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.700987 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.700992 | controller | - ' logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.700996 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.701000 | controller | - ' logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.701004 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.701009 | controller | found' 2026-03-09 19:10:03.701013 | controller | - ' logger.go:42: 18:52:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.701017 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.701021 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.701026 | controller | - ' ' 2026-03-09 19:10:03.701030 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.701037 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.701044 | controller | - ' ' 2026-03-09 19:10:03.701049 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.701053 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.701057 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.701062 | controller | - ' exit 1' 2026-03-09 19:10:03.701066 | controller | - ' fi' 2026-03-09 19:10:03.701070 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.701075 | controller | - ' ' 2026-03-09 19:10:03.701079 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.701083 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.701087 | controller | | base64 -d)' 2026-03-09 19:10:03.701098 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.701103 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.701108 | controller | - ' exit 1' 2026-03-09 19:10:03.701114 | controller | - ' fi' 2026-03-09 19:10:03.701118 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.701123 | controller | - ' ' 2026-03-09 19:10:03.701127 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.701131 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.701136 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.701141 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.701156 | controller | --timeout=300s' 2026-03-09 19:10:03.701163 | controller | - ' ' 2026-03-09 19:10:03.701167 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.701172 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.701176 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.701180 | controller | - ' ' 2026-03-09 19:10:03.701184 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701189 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701193 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701197 | controller | - ' ' 2026-03-09 19:10:03.701202 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.701208 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.701213 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.701217 | controller | available"' 2026-03-09 19:10:03.701222 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.701226 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.701230 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.701234 | controller | - ' exit 1' 2026-03-09 19:10:03.701239 | controller | - ' fi' 2026-03-09 19:10:03.701243 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.701247 | controller | - ' ' 2026-03-09 19:10:03.701252 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.701256 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.701260 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701266 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.701271 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701275 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.701279 | controller | -- \' 2026-03-09 19:10:03.701285 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701290 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.701295 | controller | - ' logger.go:42: 18:52:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.701299 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.701304 | controller | - ' logger.go:42: 18:52:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.701308 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.701312 | controller | found' 2026-03-09 19:10:03.701316 | controller | - ' logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.701321 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.701325 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.701329 | controller | - ' ' 2026-03-09 19:10:03.701334 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.701338 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.701342 | controller | - ' ' 2026-03-09 19:10:03.701346 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.701351 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.701355 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.701359 | controller | - ' exit 1' 2026-03-09 19:10:03.701364 | controller | - ' fi' 2026-03-09 19:10:03.701368 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.701372 | controller | - ' ' 2026-03-09 19:10:03.701377 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.701381 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.701385 | controller | | base64 -d)' 2026-03-09 19:10:03.701390 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.701401 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.701406 | controller | - ' exit 1' 2026-03-09 19:10:03.701410 | controller | - ' fi' 2026-03-09 19:10:03.701415 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.701419 | controller | - ' ' 2026-03-09 19:10:03.701423 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.701430 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.701434 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.701439 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.701443 | controller | --timeout=300s' 2026-03-09 19:10:03.701447 | controller | - ' ' 2026-03-09 19:10:03.701451 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.701456 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.701460 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.701466 | controller | - ' ' 2026-03-09 19:10:03.701471 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701475 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701484 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701489 | controller | - ' ' 2026-03-09 19:10:03.701493 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.701497 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.701502 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.701506 | controller | available"' 2026-03-09 19:10:03.701510 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.701514 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.701519 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.701523 | controller | - ' exit 1' 2026-03-09 19:10:03.701527 | controller | - ' fi' 2026-03-09 19:10:03.701532 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.701536 | controller | - ' ' 2026-03-09 19:10:03.701540 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.701544 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.701549 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701553 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.701557 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701561 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.701566 | controller | -- \' 2026-03-09 19:10:03.701570 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701574 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.701579 | controller | - ' logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.701583 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.701587 | controller | - ' logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.701592 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.701596 | controller | found' 2026-03-09 19:10:03.701600 | controller | - ' logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.701604 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.701609 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.701613 | controller | - ' ' 2026-03-09 19:10:03.701617 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.701621 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.701626 | controller | - ' ' 2026-03-09 19:10:03.701632 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.701636 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.701641 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.701645 | controller | - ' exit 1' 2026-03-09 19:10:03.701649 | controller | - ' fi' 2026-03-09 19:10:03.701654 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.701658 | controller | - ' ' 2026-03-09 19:10:03.701666 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.701671 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.701675 | controller | | base64 -d)' 2026-03-09 19:10:03.701679 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.701684 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.701695 | controller | - ' exit 1' 2026-03-09 19:10:03.701701 | controller | - ' fi' 2026-03-09 19:10:03.701705 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.701709 | controller | - ' ' 2026-03-09 19:10:03.701714 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.701718 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.701722 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.701726 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.701731 | controller | --timeout=300s' 2026-03-09 19:10:03.701735 | controller | - ' ' 2026-03-09 19:10:03.701739 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.701743 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.701748 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.701752 | controller | - ' ' 2026-03-09 19:10:03.701756 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701761 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701765 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.701769 | controller | - ' ' 2026-03-09 19:10:03.701773 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.701778 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.701782 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.701786 | controller | available"' 2026-03-09 19:10:03.701790 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.701795 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.701799 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.701803 | controller | - ' exit 1' 2026-03-09 19:10:03.701807 | controller | - ' fi' 2026-03-09 19:10:03.701811 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.701816 | controller | - ' ' 2026-03-09 19:10:03.701820 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.701824 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.701829 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701833 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.701837 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701841 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.701846 | controller | -- \' 2026-03-09 19:10:03.701850 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.701856 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.701860 | controller | - ' logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.701865 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.701869 | controller | - ' logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.701873 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.701878 | controller | found' 2026-03-09 19:10:03.701882 | controller | - ' logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.701886 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.701890 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.701895 | controller | - ' ' 2026-03-09 19:10:03.701899 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.701903 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.701908 | controller | - ' ' 2026-03-09 19:10:03.701912 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.701916 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.701920 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.701925 | controller | - ' exit 1' 2026-03-09 19:10:03.701929 | controller | - ' fi' 2026-03-09 19:10:03.701933 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.701938 | controller | - ' ' 2026-03-09 19:10:03.701942 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.701946 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.701950 | controller | | base64 -d)' 2026-03-09 19:10:03.701955 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.701959 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.701963 | controller | - ' exit 1' 2026-03-09 19:10:03.701967 | controller | - ' fi' 2026-03-09 19:10:03.701978 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.701983 | controller | - ' ' 2026-03-09 19:10:03.701987 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.701991 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.701996 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.702000 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.702004 | controller | --timeout=300s' 2026-03-09 19:10:03.702008 | controller | - ' ' 2026-03-09 19:10:03.702013 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.702017 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.702021 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.702026 | controller | - ' ' 2026-03-09 19:10:03.702032 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702037 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702041 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702046 | controller | - ' ' 2026-03-09 19:10:03.702050 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.702056 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.702060 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.702065 | controller | available"' 2026-03-09 19:10:03.702069 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.702073 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.702077 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.702082 | controller | - ' exit 1' 2026-03-09 19:10:03.702086 | controller | - ' fi' 2026-03-09 19:10:03.702091 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.702095 | controller | - ' ' 2026-03-09 19:10:03.702099 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.702105 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.702110 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702115 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.702119 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702125 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.702130 | controller | -- \' 2026-03-09 19:10:03.702134 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702139 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.702143 | controller | - ' logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.702160 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.702165 | controller | - ' logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.702169 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.702174 | controller | found' 2026-03-09 19:10:03.702178 | controller | - ' logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.702182 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.702186 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.702191 | controller | - ' ' 2026-03-09 19:10:03.702195 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.702199 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.702203 | controller | - ' ' 2026-03-09 19:10:03.702208 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.702212 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.702216 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.702220 | controller | - ' exit 1' 2026-03-09 19:10:03.702225 | controller | - ' fi' 2026-03-09 19:10:03.702229 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.702235 | controller | - ' ' 2026-03-09 19:10:03.702240 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.702244 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.702248 | controller | | base64 -d)' 2026-03-09 19:10:03.702253 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.702257 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.702261 | controller | - ' exit 1' 2026-03-09 19:10:03.702265 | controller | - ' fi' 2026-03-09 19:10:03.702280 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.702285 | controller | - ' ' 2026-03-09 19:10:03.702289 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.702294 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.702298 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.702302 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.702306 | controller | --timeout=300s' 2026-03-09 19:10:03.702311 | controller | - ' ' 2026-03-09 19:10:03.702315 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.702319 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.702324 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.702328 | controller | - ' ' 2026-03-09 19:10:03.702332 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702337 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702341 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702345 | controller | - ' ' 2026-03-09 19:10:03.702349 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.702354 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.702358 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.702362 | controller | available"' 2026-03-09 19:10:03.702366 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.702371 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.702375 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.702379 | controller | - ' exit 1' 2026-03-09 19:10:03.702384 | controller | - ' fi' 2026-03-09 19:10:03.702388 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.702392 | controller | - ' ' 2026-03-09 19:10:03.702396 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.702401 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.702405 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702409 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.702414 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702418 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.702422 | controller | -- \' 2026-03-09 19:10:03.702427 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702431 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.702439 | controller | - ' logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.702443 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.702447 | controller | - ' logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.702452 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.702458 | controller | found' 2026-03-09 19:10:03.702462 | controller | - ' logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.702466 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.702471 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.702475 | controller | - ' ' 2026-03-09 19:10:03.702479 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.702483 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.702488 | controller | - ' ' 2026-03-09 19:10:03.702492 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.702496 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.702501 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.702505 | controller | - ' exit 1' 2026-03-09 19:10:03.702509 | controller | - ' fi' 2026-03-09 19:10:03.702513 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.702518 | controller | - ' ' 2026-03-09 19:10:03.702522 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.702526 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.702531 | controller | | base64 -d)' 2026-03-09 19:10:03.702535 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.702539 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.702543 | controller | - ' exit 1' 2026-03-09 19:10:03.702548 | controller | - ' fi' 2026-03-09 19:10:03.702552 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.702556 | controller | - ' ' 2026-03-09 19:10:03.702567 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.702572 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.702576 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.702580 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.702585 | controller | --timeout=300s' 2026-03-09 19:10:03.702589 | controller | - ' ' 2026-03-09 19:10:03.702593 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.702597 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.702602 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.702606 | controller | - ' ' 2026-03-09 19:10:03.702610 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702614 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702619 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702623 | controller | - ' ' 2026-03-09 19:10:03.702627 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.702632 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.702636 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.702640 | controller | available"' 2026-03-09 19:10:03.702645 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.702649 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.702653 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.702657 | controller | - ' exit 1' 2026-03-09 19:10:03.702663 | controller | - ' fi' 2026-03-09 19:10:03.702668 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.702672 | controller | - ' ' 2026-03-09 19:10:03.702676 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.702680 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.702685 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702689 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.702693 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702698 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.702702 | controller | -- \' 2026-03-09 19:10:03.702706 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702710 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.702715 | controller | - ' logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.702719 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.702723 | controller | - ' logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.702727 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.702732 | controller | found' 2026-03-09 19:10:03.702736 | controller | - ' logger.go:42: 18:52:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.702740 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.702745 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.702751 | controller | - ' ' 2026-03-09 19:10:03.702755 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.702760 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.702764 | controller | - ' ' 2026-03-09 19:10:03.702768 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.702773 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.702777 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.702781 | controller | - ' exit 1' 2026-03-09 19:10:03.702786 | controller | - ' fi' 2026-03-09 19:10:03.702790 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.702795 | controller | - ' ' 2026-03-09 19:10:03.702799 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.702803 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.702808 | controller | | base64 -d)' 2026-03-09 19:10:03.702812 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.702816 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.702821 | controller | - ' exit 1' 2026-03-09 19:10:03.702825 | controller | - ' fi' 2026-03-09 19:10:03.702829 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.702834 | controller | - ' ' 2026-03-09 19:10:03.702838 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.702850 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.702856 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.702860 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.702867 | controller | --timeout=300s' 2026-03-09 19:10:03.702872 | controller | - ' ' 2026-03-09 19:10:03.702876 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.702880 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.702885 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.702889 | controller | - ' ' 2026-03-09 19:10:03.702893 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702898 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702902 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.702906 | controller | - ' ' 2026-03-09 19:10:03.702911 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.702915 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.702919 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.702924 | controller | available"' 2026-03-09 19:10:03.702928 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.702934 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.702938 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.702943 | controller | - ' exit 1' 2026-03-09 19:10:03.702947 | controller | - ' fi' 2026-03-09 19:10:03.702951 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.702956 | controller | - ' ' 2026-03-09 19:10:03.702960 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.702964 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.702968 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702973 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.702977 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702981 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.702986 | controller | -- \' 2026-03-09 19:10:03.702990 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.702994 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.702999 | controller | - ' logger.go:42: 18:52:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.703003 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.703008 | controller | - ' logger.go:42: 18:52:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.703012 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.703016 | controller | found' 2026-03-09 19:10:03.703021 | controller | - ' logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.703025 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.703029 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.703033 | controller | - ' ' 2026-03-09 19:10:03.703038 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.703042 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.703046 | controller | - ' ' 2026-03-09 19:10:03.703124 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.703129 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.703133 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.703137 | controller | - ' exit 1' 2026-03-09 19:10:03.703142 | controller | - ' fi' 2026-03-09 19:10:03.703156 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.703163 | controller | - ' ' 2026-03-09 19:10:03.703167 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.703171 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.703176 | controller | | base64 -d)' 2026-03-09 19:10:03.703180 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.703184 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.703189 | controller | - ' exit 1' 2026-03-09 19:10:03.703195 | controller | - ' fi' 2026-03-09 19:10:03.703199 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.703204 | controller | - ' ' 2026-03-09 19:10:03.703208 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.703220 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.703225 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.703229 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.703233 | controller | --timeout=300s' 2026-03-09 19:10:03.703238 | controller | - ' ' 2026-03-09 19:10:03.703242 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.703246 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.703251 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.703255 | controller | - ' ' 2026-03-09 19:10:03.703259 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703263 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703268 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703272 | controller | - ' ' 2026-03-09 19:10:03.703276 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.703281 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.703285 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.703289 | controller | available"' 2026-03-09 19:10:03.703293 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.703298 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.703302 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.703306 | controller | - ' exit 1' 2026-03-09 19:10:03.703310 | controller | - ' fi' 2026-03-09 19:10:03.703315 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.703319 | controller | - ' ' 2026-03-09 19:10:03.703323 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.703328 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.703332 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703338 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.703342 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703347 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.703351 | controller | -- \' 2026-03-09 19:10:03.703355 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703359 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.703364 | controller | - ' logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.703368 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.703372 | controller | - ' logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.703377 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.703381 | controller | found' 2026-03-09 19:10:03.703385 | controller | - ' logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.703389 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.703394 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.703398 | controller | - ' ' 2026-03-09 19:10:03.703402 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.703406 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.703411 | controller | - ' ' 2026-03-09 19:10:03.703415 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.703419 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.703423 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.703428 | controller | - ' exit 1' 2026-03-09 19:10:03.703432 | controller | - ' fi' 2026-03-09 19:10:03.703436 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.703441 | controller | - ' ' 2026-03-09 19:10:03.703445 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.703449 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.703453 | controller | | base64 -d)' 2026-03-09 19:10:03.703458 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.703462 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.703466 | controller | - ' exit 1' 2026-03-09 19:10:03.703471 | controller | - ' fi' 2026-03-09 19:10:03.703475 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.703479 | controller | - ' ' 2026-03-09 19:10:03.703483 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.703488 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.703498 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.703504 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.703509 | controller | --timeout=300s' 2026-03-09 19:10:03.703513 | controller | - ' ' 2026-03-09 19:10:03.703517 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.703522 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.703526 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.703530 | controller | - ' ' 2026-03-09 19:10:03.703535 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703541 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703545 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703549 | controller | - ' ' 2026-03-09 19:10:03.703554 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.703559 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.703564 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.703568 | controller | available"' 2026-03-09 19:10:03.703572 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.703576 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.703581 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.703585 | controller | - ' exit 1' 2026-03-09 19:10:03.703589 | controller | - ' fi' 2026-03-09 19:10:03.703594 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.703598 | controller | - ' ' 2026-03-09 19:10:03.703602 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.703606 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.703611 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703615 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.703619 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703623 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.703628 | controller | -- \' 2026-03-09 19:10:03.703632 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703636 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.703642 | controller | - ' logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.703646 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.703651 | controller | - ' logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.703655 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.703659 | controller | found' 2026-03-09 19:10:03.703664 | controller | - ' logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.703668 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.703672 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.703677 | controller | - ' ' 2026-03-09 19:10:03.703681 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.703685 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.703689 | controller | - ' ' 2026-03-09 19:10:03.703694 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.703698 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.703702 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.703706 | controller | - ' exit 1' 2026-03-09 19:10:03.703711 | controller | - ' fi' 2026-03-09 19:10:03.703715 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.703719 | controller | - ' ' 2026-03-09 19:10:03.703742 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.703752 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.703756 | controller | | base64 -d)' 2026-03-09 19:10:03.703760 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.703765 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.703769 | controller | - ' exit 1' 2026-03-09 19:10:03.703773 | controller | - ' fi' 2026-03-09 19:10:03.703778 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.703782 | controller | - ' ' 2026-03-09 19:10:03.703786 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.703790 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.703795 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.703806 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.703811 | controller | --timeout=300s' 2026-03-09 19:10:03.703818 | controller | - ' ' 2026-03-09 19:10:03.703823 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.703827 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.703832 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.703836 | controller | - ' ' 2026-03-09 19:10:03.703840 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703844 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703849 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.703853 | controller | - ' ' 2026-03-09 19:10:03.703857 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.703862 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.703866 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.703872 | controller | available"' 2026-03-09 19:10:03.703876 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.703880 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.703885 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.703889 | controller | - ' exit 1' 2026-03-09 19:10:03.703893 | controller | - ' fi' 2026-03-09 19:10:03.703897 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.703902 | controller | - ' ' 2026-03-09 19:10:03.703906 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.703910 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.703914 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703919 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.703923 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703927 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.703932 | controller | -- \' 2026-03-09 19:10:03.703936 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.703942 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.703946 | controller | - ' logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.703950 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.703955 | controller | - ' logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.703959 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.703963 | controller | found' 2026-03-09 19:10:03.703968 | controller | - ' logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.703972 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.703976 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.703980 | controller | - ' ' 2026-03-09 19:10:03.703985 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.703989 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.703993 | controller | - ' ' 2026-03-09 19:10:03.703998 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.704002 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.704006 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.704010 | controller | - ' exit 1' 2026-03-09 19:10:03.704015 | controller | - ' fi' 2026-03-09 19:10:03.704019 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.704023 | controller | - ' ' 2026-03-09 19:10:03.704028 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.704032 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.704036 | controller | | base64 -d)' 2026-03-09 19:10:03.704040 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.704045 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.704049 | controller | - ' exit 1' 2026-03-09 19:10:03.704053 | controller | - ' fi' 2026-03-09 19:10:03.704058 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.704062 | controller | - ' ' 2026-03-09 19:10:03.704066 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.704070 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.704075 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.704086 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.704091 | controller | --timeout=300s' 2026-03-09 19:10:03.704095 | controller | - ' ' 2026-03-09 19:10:03.704099 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.704104 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.704108 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.704112 | controller | - ' ' 2026-03-09 19:10:03.704117 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704121 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704125 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704129 | controller | - ' ' 2026-03-09 19:10:03.704136 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.704140 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.704158 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.704164 | controller | available"' 2026-03-09 19:10:03.704169 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.704173 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.704177 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.704201 | controller | - ' exit 1' 2026-03-09 19:10:03.704206 | controller | - ' fi' 2026-03-09 19:10:03.704210 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.704214 | controller | - ' ' 2026-03-09 19:10:03.704219 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.704223 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.704227 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704231 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.704236 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704240 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.704244 | controller | -- \' 2026-03-09 19:10:03.704248 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704253 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.704257 | controller | - ' logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.704261 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.704282 | controller | - ' logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.704286 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.704290 | controller | found' 2026-03-09 19:10:03.704294 | controller | - ' logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.704299 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.704303 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.704307 | controller | - ' ' 2026-03-09 19:10:03.704312 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.704316 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.704320 | controller | - ' ' 2026-03-09 19:10:03.704327 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.704331 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.704336 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.704340 | controller | - ' exit 1' 2026-03-09 19:10:03.704344 | controller | - ' fi' 2026-03-09 19:10:03.704348 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.704353 | controller | - ' ' 2026-03-09 19:10:03.704359 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.704363 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.704368 | controller | | base64 -d)' 2026-03-09 19:10:03.704372 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.704376 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.704381 | controller | - ' exit 1' 2026-03-09 19:10:03.704385 | controller | - ' fi' 2026-03-09 19:10:03.704389 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.704395 | controller | - ' ' 2026-03-09 19:10:03.704400 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.704412 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.704416 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.704421 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.704425 | controller | --timeout=300s' 2026-03-09 19:10:03.704436 | controller | - ' ' 2026-03-09 19:10:03.704441 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.704445 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.704450 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.704454 | controller | - ' ' 2026-03-09 19:10:03.704458 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704463 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704467 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704471 | controller | - ' ' 2026-03-09 19:10:03.704475 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.704480 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.704484 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.704488 | controller | available"' 2026-03-09 19:10:03.704493 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.704497 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.704501 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.704505 | controller | - ' exit 1' 2026-03-09 19:10:03.704510 | controller | - ' fi' 2026-03-09 19:10:03.704514 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.704518 | controller | - ' ' 2026-03-09 19:10:03.704522 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.704527 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.704531 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704535 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.704539 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704544 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.704548 | controller | -- \' 2026-03-09 19:10:03.704552 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704556 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.704561 | controller | - ' logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.704565 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.704569 | controller | - ' logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.704573 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.704578 | controller | found' 2026-03-09 19:10:03.704584 | controller | - ' logger.go:42: 18:52:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.704588 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.704593 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.704597 | controller | - ' ' 2026-03-09 19:10:03.704601 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.704606 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.704610 | controller | - ' ' 2026-03-09 19:10:03.704614 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.704619 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.704623 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.704627 | controller | - ' exit 1' 2026-03-09 19:10:03.704632 | controller | - ' fi' 2026-03-09 19:10:03.704636 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.704640 | controller | - ' ' 2026-03-09 19:10:03.704644 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.704649 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.704653 | controller | | base64 -d)' 2026-03-09 19:10:03.704657 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.704662 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.704666 | controller | - ' exit 1' 2026-03-09 19:10:03.704670 | controller | - ' fi' 2026-03-09 19:10:03.704675 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.704679 | controller | - ' ' 2026-03-09 19:10:03.704683 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.704688 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.704692 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.704696 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.704701 | controller | --timeout=300s' 2026-03-09 19:10:03.704705 | controller | - ' ' 2026-03-09 19:10:03.704716 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.704720 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.704727 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.704731 | controller | - ' ' 2026-03-09 19:10:03.704736 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704740 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704744 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.704749 | controller | - ' ' 2026-03-09 19:10:03.704753 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.704757 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.704763 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.704767 | controller | available"' 2026-03-09 19:10:03.704772 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.704776 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.704780 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.704785 | controller | - ' exit 1' 2026-03-09 19:10:03.704789 | controller | - ' fi' 2026-03-09 19:10:03.704795 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.704800 | controller | - ' ' 2026-03-09 19:10:03.704804 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.704808 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.704813 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704817 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.704821 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704826 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.704830 | controller | -- \' 2026-03-09 19:10:03.704834 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.704839 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.704843 | controller | - ' logger.go:42: 18:52:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.704847 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.704852 | controller | - ' logger.go:42: 18:52:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.704856 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.704860 | controller | found' 2026-03-09 19:10:03.704865 | controller | - ' logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.704869 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.704873 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.704878 | controller | - ' ' 2026-03-09 19:10:03.704882 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.704886 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.704891 | controller | - ' ' 2026-03-09 19:10:03.704895 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.704899 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.704904 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.704908 | controller | - ' exit 1' 2026-03-09 19:10:03.704912 | controller | - ' fi' 2026-03-09 19:10:03.704917 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.704921 | controller | - ' ' 2026-03-09 19:10:03.704925 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.704930 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.704934 | controller | | base64 -d)' 2026-03-09 19:10:03.704938 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.704943 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.704947 | controller | - ' exit 1' 2026-03-09 19:10:03.704951 | controller | - ' fi' 2026-03-09 19:10:03.704956 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.704960 | controller | - ' ' 2026-03-09 19:10:03.704964 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.704969 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.704973 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.704977 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.704982 | controller | --timeout=300s' 2026-03-09 19:10:03.704986 | controller | - ' ' 2026-03-09 19:10:03.705000 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.705004 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.705009 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.705013 | controller | - ' ' 2026-03-09 19:10:03.705019 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705024 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705028 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705033 | controller | - ' ' 2026-03-09 19:10:03.705037 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.705041 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.705046 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.705050 | controller | available"' 2026-03-09 19:10:03.705054 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.705059 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.705063 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.705067 | controller | - ' exit 1' 2026-03-09 19:10:03.705072 | controller | - ' fi' 2026-03-09 19:10:03.705076 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.705080 | controller | - ' ' 2026-03-09 19:10:03.705085 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.705089 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.705093 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705097 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.705102 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705106 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.705110 | controller | -- \' 2026-03-09 19:10:03.705115 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705119 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.705123 | controller | - ' logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.705128 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.705132 | controller | - ' logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.705136 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.705141 | controller | found' 2026-03-09 19:10:03.705162 | controller | - ' logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.705169 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.705173 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.705178 | controller | - ' ' 2026-03-09 19:10:03.705182 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.705186 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.705191 | controller | - ' ' 2026-03-09 19:10:03.705195 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.705201 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.705206 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.705210 | controller | - ' exit 1' 2026-03-09 19:10:03.705215 | controller | - ' fi' 2026-03-09 19:10:03.705219 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.705223 | controller | - ' ' 2026-03-09 19:10:03.705228 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.705232 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.705236 | controller | | base64 -d)' 2026-03-09 19:10:03.705241 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.705245 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.705249 | controller | - ' exit 1' 2026-03-09 19:10:03.705253 | controller | - ' fi' 2026-03-09 19:10:03.705258 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.705262 | controller | - ' ' 2026-03-09 19:10:03.705267 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.705271 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.705275 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.705279 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.705284 | controller | --timeout=300s' 2026-03-09 19:10:03.705288 | controller | - ' ' 2026-03-09 19:10:03.705293 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.705305 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.705310 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.705314 | controller | - ' ' 2026-03-09 19:10:03.705319 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705323 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705328 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705332 | controller | - ' ' 2026-03-09 19:10:03.705336 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.705341 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.705345 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.705349 | controller | available"' 2026-03-09 19:10:03.705358 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.705363 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.705368 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.705372 | controller | - ' exit 1' 2026-03-09 19:10:03.705376 | controller | - ' fi' 2026-03-09 19:10:03.705381 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.705385 | controller | - ' ' 2026-03-09 19:10:03.705389 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.705394 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.705398 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705402 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.705409 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705413 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.705417 | controller | -- \' 2026-03-09 19:10:03.705422 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705426 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.705430 | controller | - ' logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.705435 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.705439 | controller | - ' logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.705443 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.705447 | controller | found' 2026-03-09 19:10:03.705452 | controller | - ' logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.705456 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.705460 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.705465 | controller | - ' ' 2026-03-09 19:10:03.705469 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.705473 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.705478 | controller | - ' ' 2026-03-09 19:10:03.705482 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.705486 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.705491 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.705495 | controller | - ' exit 1' 2026-03-09 19:10:03.705499 | controller | - ' fi' 2026-03-09 19:10:03.705503 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.705508 | controller | - ' ' 2026-03-09 19:10:03.705512 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.705516 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.705521 | controller | | base64 -d)' 2026-03-09 19:10:03.705525 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.705529 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.705534 | controller | - ' exit 1' 2026-03-09 19:10:03.705538 | controller | - ' fi' 2026-03-09 19:10:03.705542 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.705547 | controller | - ' ' 2026-03-09 19:10:03.705551 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.705555 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.705559 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.705564 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.705568 | controller | --timeout=300s' 2026-03-09 19:10:03.705572 | controller | - ' ' 2026-03-09 19:10:03.705577 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.705588 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.705592 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.705597 | controller | - ' ' 2026-03-09 19:10:03.705601 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705607 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705612 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705616 | controller | - ' ' 2026-03-09 19:10:03.705620 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.705625 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.705629 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.705633 | controller | available"' 2026-03-09 19:10:03.705638 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.705644 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.705648 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.705653 | controller | - ' exit 1' 2026-03-09 19:10:03.705657 | controller | - ' fi' 2026-03-09 19:10:03.705661 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.705666 | controller | - ' ' 2026-03-09 19:10:03.705670 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.705674 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.705679 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705683 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.705689 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705693 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.705698 | controller | -- \' 2026-03-09 19:10:03.705702 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705706 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.705711 | controller | - ' logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.705715 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.705719 | controller | - ' logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.705724 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.705728 | controller | found' 2026-03-09 19:10:03.705732 | controller | - ' logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.705736 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.705741 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.705745 | controller | - ' ' 2026-03-09 19:10:03.705750 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.705754 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.705758 | controller | - ' ' 2026-03-09 19:10:03.705762 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.705767 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.705771 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.705775 | controller | - ' exit 1' 2026-03-09 19:10:03.705780 | controller | - ' fi' 2026-03-09 19:10:03.705784 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.705788 | controller | - ' ' 2026-03-09 19:10:03.705793 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.705797 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.705803 | controller | | base64 -d)' 2026-03-09 19:10:03.705807 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.705812 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.705816 | controller | - ' exit 1' 2026-03-09 19:10:03.705820 | controller | - ' fi' 2026-03-09 19:10:03.705825 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.705829 | controller | - ' ' 2026-03-09 19:10:03.705833 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.705838 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.705842 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.705846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.705850 | controller | --timeout=300s' 2026-03-09 19:10:03.705855 | controller | - ' ' 2026-03-09 19:10:03.705859 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.705863 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.705874 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.705879 | controller | - ' ' 2026-03-09 19:10:03.705883 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705887 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705892 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.705896 | controller | - ' ' 2026-03-09 19:10:03.705900 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.705905 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.705909 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.705913 | controller | available"' 2026-03-09 19:10:03.705917 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.705922 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.705926 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.705930 | controller | - ' exit 1' 2026-03-09 19:10:03.705935 | controller | - ' fi' 2026-03-09 19:10:03.705939 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.705943 | controller | - ' ' 2026-03-09 19:10:03.705949 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.705954 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.705958 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705962 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.705967 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705971 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.705975 | controller | -- \' 2026-03-09 19:10:03.705979 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.705984 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.705990 | controller | - ' logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.705994 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.705999 | controller | - ' logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.706003 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.706007 | controller | found' 2026-03-09 19:10:03.706011 | controller | - ' logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.706016 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.706020 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.706024 | controller | - ' ' 2026-03-09 19:10:03.706029 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.706033 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.706037 | controller | - ' ' 2026-03-09 19:10:03.706041 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.706045 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.706050 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.706054 | controller | - ' exit 1' 2026-03-09 19:10:03.706058 | controller | - ' fi' 2026-03-09 19:10:03.706063 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.706067 | controller | - ' ' 2026-03-09 19:10:03.706071 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.706075 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.706080 | controller | | base64 -d)' 2026-03-09 19:10:03.706084 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.706088 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.706092 | controller | - ' exit 1' 2026-03-09 19:10:03.706097 | controller | - ' fi' 2026-03-09 19:10:03.706101 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.706106 | controller | - ' ' 2026-03-09 19:10:03.706112 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.706117 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.706121 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.706126 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.706131 | controller | --timeout=300s' 2026-03-09 19:10:03.706136 | controller | - ' ' 2026-03-09 19:10:03.706140 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.706156 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.706169 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.706173 | controller | - ' ' 2026-03-09 19:10:03.706178 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706182 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706187 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706191 | controller | - ' ' 2026-03-09 19:10:03.706195 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.706199 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.706204 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.706210 | controller | available"' 2026-03-09 19:10:03.706215 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.706219 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.706223 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.706227 | controller | - ' exit 1' 2026-03-09 19:10:03.706232 | controller | - ' fi' 2026-03-09 19:10:03.706236 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.706240 | controller | - ' ' 2026-03-09 19:10:03.706244 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.706249 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.706253 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706257 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.706262 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706266 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.706270 | controller | -- \' 2026-03-09 19:10:03.706274 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706280 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.706285 | controller | - ' logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.706289 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.706293 | controller | - ' logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.706298 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.706302 | controller | found' 2026-03-09 19:10:03.706307 | controller | - ' logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.706312 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.706318 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.706323 | controller | - ' ' 2026-03-09 19:10:03.706327 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.706331 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.706335 | controller | - ' ' 2026-03-09 19:10:03.706340 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.706344 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.706348 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.706353 | controller | - ' exit 1' 2026-03-09 19:10:03.706357 | controller | - ' fi' 2026-03-09 19:10:03.706363 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.706369 | controller | - ' ' 2026-03-09 19:10:03.706374 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.706380 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.706385 | controller | | base64 -d)' 2026-03-09 19:10:03.706391 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.706400 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.706405 | controller | - ' exit 1' 2026-03-09 19:10:03.706409 | controller | - ' fi' 2026-03-09 19:10:03.706413 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.706418 | controller | - ' ' 2026-03-09 19:10:03.706425 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.706431 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.706437 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.706443 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.706449 | controller | --timeout=300s' 2026-03-09 19:10:03.706455 | controller | - ' ' 2026-03-09 19:10:03.706461 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.706467 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.706472 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.706489 | controller | - ' ' 2026-03-09 19:10:03.706496 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706501 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706507 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706512 | controller | - ' ' 2026-03-09 19:10:03.706518 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.706523 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.706529 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.706534 | controller | available"' 2026-03-09 19:10:03.706540 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.706545 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.706551 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.706557 | controller | - ' exit 1' 2026-03-09 19:10:03.706563 | controller | - ' fi' 2026-03-09 19:10:03.706569 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.706574 | controller | - ' ' 2026-03-09 19:10:03.706578 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.706582 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.706587 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706594 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.706598 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706603 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.706607 | controller | -- \' 2026-03-09 19:10:03.706611 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706615 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.706620 | controller | - ' logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.706624 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.706631 | controller | - ' logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.706636 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.706640 | controller | found' 2026-03-09 19:10:03.706644 | controller | - ' logger.go:42: 18:53:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.706651 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.706655 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.706660 | controller | - ' ' 2026-03-09 19:10:03.706664 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.706668 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.706673 | controller | - ' ' 2026-03-09 19:10:03.706677 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.706681 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.706685 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.706690 | controller | - ' exit 1' 2026-03-09 19:10:03.706694 | controller | - ' fi' 2026-03-09 19:10:03.706698 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.706702 | controller | - ' ' 2026-03-09 19:10:03.706707 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.706711 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.706715 | controller | | base64 -d)' 2026-03-09 19:10:03.706719 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.706724 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.706728 | controller | - ' exit 1' 2026-03-09 19:10:03.706732 | controller | - ' fi' 2026-03-09 19:10:03.706737 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.706741 | controller | - ' ' 2026-03-09 19:10:03.706745 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.706749 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.706754 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.706758 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.706762 | controller | --timeout=300s' 2026-03-09 19:10:03.706766 | controller | - ' ' 2026-03-09 19:10:03.706771 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.706775 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.706779 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.706783 | controller | - ' ' 2026-03-09 19:10:03.706797 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706803 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706808 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.706812 | controller | - ' ' 2026-03-09 19:10:03.706816 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.706821 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.706826 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.706832 | controller | available"' 2026-03-09 19:10:03.706838 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.706844 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.706850 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.706856 | controller | - ' exit 1' 2026-03-09 19:10:03.706861 | controller | - ' fi' 2026-03-09 19:10:03.706866 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.706872 | controller | - ' ' 2026-03-09 19:10:03.706877 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.706883 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.706888 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706892 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.706896 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706901 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.706905 | controller | -- \' 2026-03-09 19:10:03.706909 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.706914 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.706918 | controller | - ' logger.go:42: 18:53:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.706922 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.706926 | controller | - ' logger.go:42: 18:53:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.706931 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.706935 | controller | found' 2026-03-09 19:10:03.706939 | controller | - ' logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.706943 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.706948 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.706952 | controller | - ' ' 2026-03-09 19:10:03.706956 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.706961 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.706965 | controller | - ' ' 2026-03-09 19:10:03.706969 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.706974 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.706978 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.706982 | controller | - ' exit 1' 2026-03-09 19:10:03.706986 | controller | - ' fi' 2026-03-09 19:10:03.706991 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.706995 | controller | - ' ' 2026-03-09 19:10:03.707001 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.707006 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.707010 | controller | | base64 -d)' 2026-03-09 19:10:03.707015 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.707019 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.707023 | controller | - ' exit 1' 2026-03-09 19:10:03.707027 | controller | - ' fi' 2026-03-09 19:10:03.707032 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.707036 | controller | - ' ' 2026-03-09 19:10:03.707040 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.707045 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.707051 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.707056 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.707060 | controller | --timeout=300s' 2026-03-09 19:10:03.707064 | controller | - ' ' 2026-03-09 19:10:03.707069 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.707075 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.707079 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.707083 | controller | - ' ' 2026-03-09 19:10:03.707097 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707101 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707106 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707111 | controller | - ' ' 2026-03-09 19:10:03.707117 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.707121 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.707125 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.707130 | controller | available"' 2026-03-09 19:10:03.707134 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.707138 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.707143 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.707166 | controller | - ' exit 1' 2026-03-09 19:10:03.707171 | controller | - ' fi' 2026-03-09 19:10:03.707175 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.707179 | controller | - ' ' 2026-03-09 19:10:03.707184 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.707188 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.707192 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707196 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.707202 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707208 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.707212 | controller | -- \' 2026-03-09 19:10:03.707216 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707223 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.707227 | controller | - ' logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.707231 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.707236 | controller | - ' logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.707240 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.707244 | controller | found' 2026-03-09 19:10:03.707248 | controller | - ' logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.707253 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.707257 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.707261 | controller | - ' ' 2026-03-09 19:10:03.707266 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.707270 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.707274 | controller | - ' ' 2026-03-09 19:10:03.707280 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.707287 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.707292 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.707296 | controller | - ' exit 1' 2026-03-09 19:10:03.707300 | controller | - ' fi' 2026-03-09 19:10:03.707304 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.707309 | controller | - ' ' 2026-03-09 19:10:03.707313 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.707317 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.707322 | controller | | base64 -d)' 2026-03-09 19:10:03.707326 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.707330 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.707334 | controller | - ' exit 1' 2026-03-09 19:10:03.707339 | controller | - ' fi' 2026-03-09 19:10:03.707343 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.707347 | controller | - ' ' 2026-03-09 19:10:03.707352 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.707356 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.707360 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.707364 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.707369 | controller | --timeout=300s' 2026-03-09 19:10:03.707373 | controller | - ' ' 2026-03-09 19:10:03.707377 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.707382 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.707386 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.707390 | controller | - ' ' 2026-03-09 19:10:03.707394 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707407 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707411 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707416 | controller | - ' ' 2026-03-09 19:10:03.707420 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.707425 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.707429 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.707433 | controller | available"' 2026-03-09 19:10:03.707438 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.707442 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.707446 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.707450 | controller | - ' exit 1' 2026-03-09 19:10:03.707455 | controller | - ' fi' 2026-03-09 19:10:03.707459 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.707463 | controller | - ' ' 2026-03-09 19:10:03.707468 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.707472 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.707476 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707480 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.707487 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707491 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.707495 | controller | -- \' 2026-03-09 19:10:03.707499 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707504 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.707508 | controller | - ' logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.707512 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.707517 | controller | - ' logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.707521 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.707525 | controller | found' 2026-03-09 19:10:03.707530 | controller | - ' logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.707535 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.707541 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.707545 | controller | - ' ' 2026-03-09 19:10:03.707549 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.707554 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.707558 | controller | - ' ' 2026-03-09 19:10:03.707562 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.707566 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.707571 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.707575 | controller | - ' exit 1' 2026-03-09 19:10:03.707579 | controller | - ' fi' 2026-03-09 19:10:03.707586 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.707592 | controller | - ' ' 2026-03-09 19:10:03.707597 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.707601 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.707606 | controller | | base64 -d)' 2026-03-09 19:10:03.707610 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.707614 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.707618 | controller | - ' exit 1' 2026-03-09 19:10:03.707623 | controller | - ' fi' 2026-03-09 19:10:03.707627 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.707631 | controller | - ' ' 2026-03-09 19:10:03.707635 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.707640 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.707645 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.707650 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.707656 | controller | --timeout=300s' 2026-03-09 19:10:03.707661 | controller | - ' ' 2026-03-09 19:10:03.707665 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.707669 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.707674 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.707678 | controller | - ' ' 2026-03-09 19:10:03.707682 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707694 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707701 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707705 | controller | - ' ' 2026-03-09 19:10:03.707709 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.707714 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.707737 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.707743 | controller | available"' 2026-03-09 19:10:03.707748 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.707753 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.707757 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.707761 | controller | - ' exit 1' 2026-03-09 19:10:03.707766 | controller | - ' fi' 2026-03-09 19:10:03.707770 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.707774 | controller | - ' ' 2026-03-09 19:10:03.707778 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.707783 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.707787 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707791 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.707795 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707800 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.707806 | controller | -- \' 2026-03-09 19:10:03.707811 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.707815 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.707821 | controller | - ' logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.707825 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.707831 | controller | - ' logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.707835 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.707839 | controller | found' 2026-03-09 19:10:03.707844 | controller | - ' logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.707848 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.707852 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.707856 | controller | - ' ' 2026-03-09 19:10:03.707861 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.707865 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.707869 | controller | - ' ' 2026-03-09 19:10:03.707873 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.707878 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.707882 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.707886 | controller | - ' exit 1' 2026-03-09 19:10:03.707891 | controller | - ' fi' 2026-03-09 19:10:03.707895 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.707899 | controller | - ' ' 2026-03-09 19:10:03.707903 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.707908 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.707914 | controller | | base64 -d)' 2026-03-09 19:10:03.707918 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.707922 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.707927 | controller | - ' exit 1' 2026-03-09 19:10:03.707931 | controller | - ' fi' 2026-03-09 19:10:03.707935 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.707940 | controller | - ' ' 2026-03-09 19:10:03.707944 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.707948 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.707952 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.707957 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.707961 | controller | --timeout=300s' 2026-03-09 19:10:03.707968 | controller | - ' ' 2026-03-09 19:10:03.707973 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.707977 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.707981 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.707985 | controller | - ' ' 2026-03-09 19:10:03.707990 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.707994 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708006 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708010 | controller | - ' ' 2026-03-09 19:10:03.708015 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.708020 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.708026 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.708030 | controller | available"' 2026-03-09 19:10:03.708034 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.708039 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.708043 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.708047 | controller | - ' exit 1' 2026-03-09 19:10:03.708051 | controller | - ' fi' 2026-03-09 19:10:03.708056 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.708060 | controller | - ' ' 2026-03-09 19:10:03.708064 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.708068 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.708073 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708077 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.708081 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708085 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.708090 | controller | -- \' 2026-03-09 19:10:03.708094 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708098 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.708102 | controller | - ' logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.708109 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.708113 | controller | - ' logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.708117 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.708122 | controller | found' 2026-03-09 19:10:03.708126 | controller | - ' logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.708130 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.708134 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.708139 | controller | - ' ' 2026-03-09 19:10:03.708143 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.708164 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.708169 | controller | - ' ' 2026-03-09 19:10:03.708173 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.708177 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.708182 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.708186 | controller | - ' exit 1' 2026-03-09 19:10:03.708190 | controller | - ' fi' 2026-03-09 19:10:03.708195 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.708199 | controller | - ' ' 2026-03-09 19:10:03.708203 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.708207 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.708212 | controller | | base64 -d)' 2026-03-09 19:10:03.708216 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.708220 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.708224 | controller | - ' exit 1' 2026-03-09 19:10:03.708229 | controller | - ' fi' 2026-03-09 19:10:03.708233 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.708237 | controller | - ' ' 2026-03-09 19:10:03.708242 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.708246 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.708250 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.708254 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.708259 | controller | --timeout=300s' 2026-03-09 19:10:03.708263 | controller | - ' ' 2026-03-09 19:10:03.708267 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.708272 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.708276 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.708280 | controller | - ' ' 2026-03-09 19:10:03.708284 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708289 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708300 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708304 | controller | - ' ' 2026-03-09 19:10:03.708309 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.708313 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.708317 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.708324 | controller | available"' 2026-03-09 19:10:03.708328 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.708335 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.708339 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.708343 | controller | - ' exit 1' 2026-03-09 19:10:03.708348 | controller | - ' fi' 2026-03-09 19:10:03.708352 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.708356 | controller | - ' ' 2026-03-09 19:10:03.708361 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.708365 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.708369 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708373 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.708378 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708382 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.708386 | controller | -- \' 2026-03-09 19:10:03.708390 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708395 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.708399 | controller | - ' logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.708403 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.708407 | controller | - ' logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.708413 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.708418 | controller | found' 2026-03-09 19:10:03.708422 | controller | - ' logger.go:42: 18:53:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.708426 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.708431 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.708435 | controller | - ' ' 2026-03-09 19:10:03.708439 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.708443 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.708448 | controller | - ' ' 2026-03-09 19:10:03.708452 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.708456 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.708461 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.708465 | controller | - ' exit 1' 2026-03-09 19:10:03.708469 | controller | - ' fi' 2026-03-09 19:10:03.708473 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.708478 | controller | - ' ' 2026-03-09 19:10:03.708482 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.708486 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.708490 | controller | | base64 -d)' 2026-03-09 19:10:03.708495 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.708499 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.708503 | controller | - ' exit 1' 2026-03-09 19:10:03.708508 | controller | - ' fi' 2026-03-09 19:10:03.708512 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.708516 | controller | - ' ' 2026-03-09 19:10:03.708520 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.708527 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.708532 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.708536 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.708540 | controller | --timeout=300s' 2026-03-09 19:10:03.708545 | controller | - ' ' 2026-03-09 19:10:03.708549 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.708553 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.708557 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.708562 | controller | - ' ' 2026-03-09 19:10:03.708566 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708570 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708575 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708579 | controller | - ' ' 2026-03-09 19:10:03.708590 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.708594 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.708599 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.708603 | controller | available"' 2026-03-09 19:10:03.708607 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.708612 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.708616 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.708620 | controller | - ' exit 1' 2026-03-09 19:10:03.708625 | controller | - ' fi' 2026-03-09 19:10:03.708631 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.708635 | controller | - ' ' 2026-03-09 19:10:03.708639 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.708643 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.708648 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708652 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.708656 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708660 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.708665 | controller | -- \' 2026-03-09 19:10:03.708669 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708673 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.708678 | controller | - ' logger.go:42: 18:53:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.708682 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.708686 | controller | - ' logger.go:42: 18:53:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.708690 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.708695 | controller | found' 2026-03-09 19:10:03.708699 | controller | - ' logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.708703 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.708710 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.708714 | controller | - ' ' 2026-03-09 19:10:03.708718 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.708723 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.708727 | controller | - ' ' 2026-03-09 19:10:03.708731 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.708735 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.708740 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.708746 | controller | - ' exit 1' 2026-03-09 19:10:03.708750 | controller | - ' fi' 2026-03-09 19:10:03.708754 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.708759 | controller | - ' ' 2026-03-09 19:10:03.708763 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.708767 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.708771 | controller | | base64 -d)' 2026-03-09 19:10:03.708776 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.708780 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.708784 | controller | - ' exit 1' 2026-03-09 19:10:03.708789 | controller | - ' fi' 2026-03-09 19:10:03.708793 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.708797 | controller | - ' ' 2026-03-09 19:10:03.708801 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.708806 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.708810 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.708814 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.708818 | controller | --timeout=300s' 2026-03-09 19:10:03.708823 | controller | - ' ' 2026-03-09 19:10:03.708827 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.708831 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.708836 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.708840 | controller | - ' ' 2026-03-09 19:10:03.708844 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708848 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708854 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.708859 | controller | - ' ' 2026-03-09 19:10:03.708870 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.708875 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.708879 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.708884 | controller | available"' 2026-03-09 19:10:03.708888 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.708892 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.708896 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.708901 | controller | - ' exit 1' 2026-03-09 19:10:03.708905 | controller | - ' fi' 2026-03-09 19:10:03.708909 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.708914 | controller | - ' ' 2026-03-09 19:10:03.708920 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.708924 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.708928 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708932 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.708937 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708941 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.708945 | controller | -- \' 2026-03-09 19:10:03.708949 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.708954 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.708958 | controller | - ' logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.708962 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.708967 | controller | - ' logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.708971 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.708975 | controller | found' 2026-03-09 19:10:03.708981 | controller | - ' logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.708986 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.708990 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.708994 | controller | - ' ' 2026-03-09 19:10:03.708998 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.709003 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.709007 | controller | - ' ' 2026-03-09 19:10:03.709011 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.709015 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.709020 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.709024 | controller | - ' exit 1' 2026-03-09 19:10:03.709028 | controller | - ' fi' 2026-03-09 19:10:03.709033 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.709037 | controller | - ' ' 2026-03-09 19:10:03.709041 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.709045 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.709050 | controller | | base64 -d)' 2026-03-09 19:10:03.709054 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.709058 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.709062 | controller | - ' exit 1' 2026-03-09 19:10:03.709067 | controller | - ' fi' 2026-03-09 19:10:03.709071 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.709075 | controller | - ' ' 2026-03-09 19:10:03.709079 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.709084 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.709088 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.709092 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.709096 | controller | --timeout=300s' 2026-03-09 19:10:03.709101 | controller | - ' ' 2026-03-09 19:10:03.709105 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.709111 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.709116 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.709120 | controller | - ' ' 2026-03-09 19:10:03.709126 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709130 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709134 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709138 | controller | - ' ' 2026-03-09 19:10:03.709143 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.709174 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.709179 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.709183 | controller | available"' 2026-03-09 19:10:03.709188 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.709192 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.709196 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.709200 | controller | - ' exit 1' 2026-03-09 19:10:03.709205 | controller | - ' fi' 2026-03-09 19:10:03.709209 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.709213 | controller | - ' ' 2026-03-09 19:10:03.709218 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.709222 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.709226 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709230 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.709235 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709239 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.709243 | controller | -- \' 2026-03-09 19:10:03.709248 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709252 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.709256 | controller | - ' logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.709260 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.709265 | controller | - ' logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.709269 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.709273 | controller | found' 2026-03-09 19:10:03.709277 | controller | - ' logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.709282 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.709286 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.709290 | controller | - ' ' 2026-03-09 19:10:03.709295 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.709299 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.709303 | controller | - ' ' 2026-03-09 19:10:03.709307 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.709312 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.709318 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.709322 | controller | - ' exit 1' 2026-03-09 19:10:03.709327 | controller | - ' fi' 2026-03-09 19:10:03.709331 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.709335 | controller | - ' ' 2026-03-09 19:10:03.709340 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.709344 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.709348 | controller | | base64 -d)' 2026-03-09 19:10:03.709352 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.709357 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.709361 | controller | - ' exit 1' 2026-03-09 19:10:03.709365 | controller | - ' fi' 2026-03-09 19:10:03.709369 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.709374 | controller | - ' ' 2026-03-09 19:10:03.709378 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.709382 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.709386 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.709391 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.709395 | controller | --timeout=300s' 2026-03-09 19:10:03.709399 | controller | - ' ' 2026-03-09 19:10:03.709404 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.709408 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.709412 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.709416 | controller | - ' ' 2026-03-09 19:10:03.709421 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709425 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709429 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709434 | controller | - ' ' 2026-03-09 19:10:03.709438 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.709442 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.709453 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.709457 | controller | available"' 2026-03-09 19:10:03.709462 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.709466 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.709470 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.709475 | controller | - ' exit 1' 2026-03-09 19:10:03.709479 | controller | - ' fi' 2026-03-09 19:10:03.709483 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.709487 | controller | - ' ' 2026-03-09 19:10:03.709492 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.709496 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.709500 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709504 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.709509 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709517 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.709521 | controller | -- \' 2026-03-09 19:10:03.709525 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709530 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.709534 | controller | - ' logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.709538 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.709542 | controller | - ' logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.709547 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.709551 | controller | found' 2026-03-09 19:10:03.709555 | controller | - ' logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.709560 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.709564 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.709568 | controller | - ' ' 2026-03-09 19:10:03.709575 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.709580 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.709584 | controller | - ' ' 2026-03-09 19:10:03.709588 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.709593 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.709597 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.709601 | controller | - ' exit 1' 2026-03-09 19:10:03.709605 | controller | - ' fi' 2026-03-09 19:10:03.709610 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.709614 | controller | - ' ' 2026-03-09 19:10:03.709620 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.709624 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.709628 | controller | | base64 -d)' 2026-03-09 19:10:03.709633 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.709637 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.709641 | controller | - ' exit 1' 2026-03-09 19:10:03.709645 | controller | - ' fi' 2026-03-09 19:10:03.709650 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.709654 | controller | - ' ' 2026-03-09 19:10:03.709658 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.709663 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.709667 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.709671 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.709675 | controller | --timeout=300s' 2026-03-09 19:10:03.709680 | controller | - ' ' 2026-03-09 19:10:03.709686 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.709690 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.709694 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.709698 | controller | - ' ' 2026-03-09 19:10:03.709703 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709707 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709713 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709718 | controller | - ' ' 2026-03-09 19:10:03.709722 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.709726 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.709738 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.709742 | controller | available"' 2026-03-09 19:10:03.709747 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.709751 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.709755 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.709759 | controller | - ' exit 1' 2026-03-09 19:10:03.709764 | controller | - ' fi' 2026-03-09 19:10:03.709768 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.709772 | controller | - ' ' 2026-03-09 19:10:03.709777 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.709781 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.709785 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709789 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.709794 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709798 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.709802 | controller | -- \' 2026-03-09 19:10:03.709806 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.709811 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.709817 | controller | - ' logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.709821 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.709826 | controller | - ' logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.709830 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.709834 | controller | found' 2026-03-09 19:10:03.709838 | controller | - ' logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.709843 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.709847 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.709851 | controller | - ' ' 2026-03-09 19:10:03.709856 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.709860 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.709864 | controller | - ' ' 2026-03-09 19:10:03.709868 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.709873 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.709877 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.709881 | controller | - ' exit 1' 2026-03-09 19:10:03.709885 | controller | - ' fi' 2026-03-09 19:10:03.709890 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.709894 | controller | - ' ' 2026-03-09 19:10:03.709898 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.709903 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.709907 | controller | | base64 -d)' 2026-03-09 19:10:03.709913 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.709917 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.709921 | controller | - ' exit 1' 2026-03-09 19:10:03.709926 | controller | - ' fi' 2026-03-09 19:10:03.709930 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.709934 | controller | - ' ' 2026-03-09 19:10:03.709939 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.709943 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.709947 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.709951 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.709956 | controller | --timeout=300s' 2026-03-09 19:10:03.709960 | controller | - ' ' 2026-03-09 19:10:03.709964 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.709969 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.709973 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.709977 | controller | - ' ' 2026-03-09 19:10:03.709981 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709986 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709990 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.709994 | controller | - ' ' 2026-03-09 19:10:03.709998 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.710003 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.710007 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.710018 | controller | available"' 2026-03-09 19:10:03.710022 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.710027 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.710031 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.710035 | controller | - ' exit 1' 2026-03-09 19:10:03.710039 | controller | - ' fi' 2026-03-09 19:10:03.710044 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.710048 | controller | - ' ' 2026-03-09 19:10:03.710052 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.710056 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.710061 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710065 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.710069 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710073 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.710078 | controller | -- \' 2026-03-09 19:10:03.710082 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710086 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.710090 | controller | - ' logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.710097 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.710101 | controller | - ' logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.710105 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.710109 | controller | found' 2026-03-09 19:10:03.710114 | controller | - ' logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.710118 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.710122 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.710129 | controller | - ' ' 2026-03-09 19:10:03.710133 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.710137 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.710143 | controller | - ' ' 2026-03-09 19:10:03.710161 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.710166 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.710170 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.710174 | controller | - ' exit 1' 2026-03-09 19:10:03.710179 | controller | - ' fi' 2026-03-09 19:10:03.710183 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.710188 | controller | - ' ' 2026-03-09 19:10:03.710192 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.710196 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.710200 | controller | | base64 -d)' 2026-03-09 19:10:03.710205 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.710209 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.710213 | controller | - ' exit 1' 2026-03-09 19:10:03.710218 | controller | - ' fi' 2026-03-09 19:10:03.710222 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.710226 | controller | - ' ' 2026-03-09 19:10:03.710231 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.710235 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.710239 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.710243 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.710248 | controller | --timeout=300s' 2026-03-09 19:10:03.710252 | controller | - ' ' 2026-03-09 19:10:03.710256 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.710260 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.710265 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.710269 | controller | - ' ' 2026-03-09 19:10:03.710273 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710278 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710282 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710286 | controller | - ' ' 2026-03-09 19:10:03.710290 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.710295 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.710299 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.710303 | controller | available"' 2026-03-09 19:10:03.710317 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.710322 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.710327 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.710331 | controller | - ' exit 1' 2026-03-09 19:10:03.710335 | controller | - ' fi' 2026-03-09 19:10:03.710340 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.710344 | controller | - ' ' 2026-03-09 19:10:03.710348 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.710353 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.710357 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710361 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.710365 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710370 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.710374 | controller | -- \' 2026-03-09 19:10:03.710378 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710384 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.710390 | controller | - ' logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.710397 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.710403 | controller | - ' logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.710409 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.710415 | controller | found' 2026-03-09 19:10:03.710419 | controller | - ' logger.go:42: 18:53:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.710423 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.710427 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.710432 | controller | - ' ' 2026-03-09 19:10:03.710436 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.710440 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.710445 | controller | - ' ' 2026-03-09 19:10:03.710449 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.710453 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.710457 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.710462 | controller | - ' exit 1' 2026-03-09 19:10:03.710466 | controller | - ' fi' 2026-03-09 19:10:03.710470 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.710474 | controller | - ' ' 2026-03-09 19:10:03.710479 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.710483 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.710487 | controller | | base64 -d)' 2026-03-09 19:10:03.710491 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.710499 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.710503 | controller | - ' exit 1' 2026-03-09 19:10:03.710507 | controller | - ' fi' 2026-03-09 19:10:03.710511 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.710516 | controller | - ' ' 2026-03-09 19:10:03.710520 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.710524 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.710531 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.710535 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.710539 | controller | --timeout=300s' 2026-03-09 19:10:03.710543 | controller | - ' ' 2026-03-09 19:10:03.710548 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.710552 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.710556 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.710561 | controller | - ' ' 2026-03-09 19:10:03.710565 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710569 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710573 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710578 | controller | - ' ' 2026-03-09 19:10:03.710582 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.710586 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.710590 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.710594 | controller | available"' 2026-03-09 19:10:03.710599 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.710611 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.710616 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.710620 | controller | - ' exit 1' 2026-03-09 19:10:03.710625 | controller | - ' fi' 2026-03-09 19:10:03.710629 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.710633 | controller | - ' ' 2026-03-09 19:10:03.710638 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.710642 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.710646 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710650 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.710655 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710659 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.710663 | controller | -- \' 2026-03-09 19:10:03.710668 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710672 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.710676 | controller | - ' logger.go:42: 18:53:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.710680 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.710685 | controller | - ' logger.go:42: 18:53:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.710689 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.710693 | controller | found' 2026-03-09 19:10:03.710698 | controller | - ' logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.710702 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.710706 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.710712 | controller | - ' ' 2026-03-09 19:10:03.710717 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.710721 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.710725 | controller | - ' ' 2026-03-09 19:10:03.710731 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.710735 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.710740 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.710744 | controller | - ' exit 1' 2026-03-09 19:10:03.710748 | controller | - ' fi' 2026-03-09 19:10:03.710753 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.710757 | controller | - ' ' 2026-03-09 19:10:03.710761 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.710766 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.710770 | controller | | base64 -d)' 2026-03-09 19:10:03.710777 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.710782 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.710786 | controller | - ' exit 1' 2026-03-09 19:10:03.710790 | controller | - ' fi' 2026-03-09 19:10:03.710794 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.710799 | controller | - ' ' 2026-03-09 19:10:03.710803 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.710807 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.710811 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.710816 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.710820 | controller | --timeout=300s' 2026-03-09 19:10:03.710824 | controller | - ' ' 2026-03-09 19:10:03.710829 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.710836 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.710842 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.710848 | controller | - ' ' 2026-03-09 19:10:03.710854 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710859 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710863 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.710867 | controller | - ' ' 2026-03-09 19:10:03.710872 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.710878 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.710882 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.710887 | controller | available"' 2026-03-09 19:10:03.710891 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.710895 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.710908 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.710912 | controller | - ' exit 1' 2026-03-09 19:10:03.710917 | controller | - ' fi' 2026-03-09 19:10:03.710921 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.710927 | controller | - ' ' 2026-03-09 19:10:03.710931 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.710938 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.710942 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710946 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.710950 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710955 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.710959 | controller | -- \' 2026-03-09 19:10:03.710963 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.710967 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.710972 | controller | - ' logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.710976 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.710980 | controller | - ' logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.710985 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.710989 | controller | found' 2026-03-09 19:10:03.710993 | controller | - ' logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.710997 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.711002 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.711007 | controller | - ' ' 2026-03-09 19:10:03.711013 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.711019 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.711025 | controller | - ' ' 2026-03-09 19:10:03.711031 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.711037 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.711042 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.711048 | controller | - ' exit 1' 2026-03-09 19:10:03.711053 | controller | - ' fi' 2026-03-09 19:10:03.711059 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.711065 | controller | - ' ' 2026-03-09 19:10:03.711071 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.711075 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.711079 | controller | | base64 -d)' 2026-03-09 19:10:03.711084 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.711088 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.711092 | controller | - ' exit 1' 2026-03-09 19:10:03.711096 | controller | - ' fi' 2026-03-09 19:10:03.711101 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.711105 | controller | - ' ' 2026-03-09 19:10:03.711109 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.711114 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.711118 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.711124 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.711129 | controller | --timeout=300s' 2026-03-09 19:10:03.711133 | controller | - ' ' 2026-03-09 19:10:03.711137 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.711142 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.711160 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.711167 | controller | - ' ' 2026-03-09 19:10:03.711171 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711176 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711180 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711184 | controller | - ' ' 2026-03-09 19:10:03.711189 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.711193 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.711197 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.711201 | controller | available"' 2026-03-09 19:10:03.711206 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.711210 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.711223 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.711228 | controller | - ' exit 1' 2026-03-09 19:10:03.711232 | controller | - ' fi' 2026-03-09 19:10:03.711237 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.711241 | controller | - ' ' 2026-03-09 19:10:03.711245 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.711250 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.711254 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711258 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.711262 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711267 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.711271 | controller | -- \' 2026-03-09 19:10:03.711275 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711280 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.711284 | controller | - ' logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.711288 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.711292 | controller | - ' logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.711297 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.711301 | controller | found' 2026-03-09 19:10:03.711305 | controller | - ' logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.711310 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.711314 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.711318 | controller | - ' ' 2026-03-09 19:10:03.711322 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.711327 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.711331 | controller | - ' ' 2026-03-09 19:10:03.711335 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.711340 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.711344 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.711350 | controller | - ' exit 1' 2026-03-09 19:10:03.711354 | controller | - ' fi' 2026-03-09 19:10:03.711359 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.711365 | controller | - ' ' 2026-03-09 19:10:03.711369 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.711373 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.711378 | controller | | base64 -d)' 2026-03-09 19:10:03.711382 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.711386 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.711391 | controller | - ' exit 1' 2026-03-09 19:10:03.711395 | controller | - ' fi' 2026-03-09 19:10:03.711399 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.711403 | controller | - ' ' 2026-03-09 19:10:03.711408 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.711412 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.711416 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.711421 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.711425 | controller | --timeout=300s' 2026-03-09 19:10:03.711429 | controller | - ' ' 2026-03-09 19:10:03.711433 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.711438 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.711442 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.711446 | controller | - ' ' 2026-03-09 19:10:03.711450 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711455 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711459 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711463 | controller | - ' ' 2026-03-09 19:10:03.711467 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.711472 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.711476 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.711480 | controller | available"' 2026-03-09 19:10:03.711484 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.711489 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.711493 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.711497 | controller | - ' exit 1' 2026-03-09 19:10:03.711501 | controller | - ' fi' 2026-03-09 19:10:03.711512 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.711517 | controller | - ' ' 2026-03-09 19:10:03.711521 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.711525 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.711530 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711534 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.711538 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711544 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.711549 | controller | -- \' 2026-03-09 19:10:03.711554 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711558 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.711563 | controller | - ' logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.711567 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.711571 | controller | - ' logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.711575 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.711580 | controller | found' 2026-03-09 19:10:03.711584 | controller | - ' logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.711590 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.711596 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.711600 | controller | - ' ' 2026-03-09 19:10:03.711605 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.711609 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.711613 | controller | - ' ' 2026-03-09 19:10:03.711618 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.711622 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.711626 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.711630 | controller | - ' exit 1' 2026-03-09 19:10:03.711635 | controller | - ' fi' 2026-03-09 19:10:03.711639 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.711643 | controller | - ' ' 2026-03-09 19:10:03.711647 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.711652 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.711656 | controller | | base64 -d)' 2026-03-09 19:10:03.711660 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.711664 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.711669 | controller | - ' exit 1' 2026-03-09 19:10:03.711673 | controller | - ' fi' 2026-03-09 19:10:03.711677 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.711682 | controller | - ' ' 2026-03-09 19:10:03.711686 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.711690 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.711694 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.711699 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.711703 | controller | --timeout=300s' 2026-03-09 19:10:03.711707 | controller | - ' ' 2026-03-09 19:10:03.711711 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.711737 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.711742 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.711746 | controller | - ' ' 2026-03-09 19:10:03.711750 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711754 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711759 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.711765 | controller | - ' ' 2026-03-09 19:10:03.711769 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.711774 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.711778 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.711782 | controller | available"' 2026-03-09 19:10:03.711786 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.711791 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.711795 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.711799 | controller | - ' exit 1' 2026-03-09 19:10:03.711804 | controller | - ' fi' 2026-03-09 19:10:03.711816 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.711820 | controller | - ' ' 2026-03-09 19:10:03.711824 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.711829 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.711833 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711837 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.711842 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711846 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.711850 | controller | -- \' 2026-03-09 19:10:03.711855 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.711859 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.711865 | controller | - ' logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.711870 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.711874 | controller | - ' logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.711878 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.711883 | controller | found' 2026-03-09 19:10:03.711887 | controller | - ' logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.711891 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.711895 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.711900 | controller | - ' ' 2026-03-09 19:10:03.711906 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.711910 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.711914 | controller | - ' ' 2026-03-09 19:10:03.711919 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.711923 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.711927 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.711932 | controller | - ' exit 1' 2026-03-09 19:10:03.711936 | controller | - ' fi' 2026-03-09 19:10:03.711940 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.711945 | controller | - ' ' 2026-03-09 19:10:03.711949 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.711953 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.711958 | controller | | base64 -d)' 2026-03-09 19:10:03.711962 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.711968 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.711972 | controller | - ' exit 1' 2026-03-09 19:10:03.711977 | controller | - ' fi' 2026-03-09 19:10:03.711981 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.711985 | controller | - ' ' 2026-03-09 19:10:03.711990 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.711994 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.712000 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.712004 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.712009 | controller | --timeout=300s' 2026-03-09 19:10:03.712014 | controller | - ' ' 2026-03-09 19:10:03.712019 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.712023 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.712027 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.712032 | controller | - ' ' 2026-03-09 19:10:03.712036 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712040 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712044 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712049 | controller | - ' ' 2026-03-09 19:10:03.712053 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.712057 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.712062 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.712066 | controller | available"' 2026-03-09 19:10:03.712070 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.712075 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.712079 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.712083 | controller | - ' exit 1' 2026-03-09 19:10:03.712091 | controller | - ' fi' 2026-03-09 19:10:03.712095 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.712100 | controller | - ' ' 2026-03-09 19:10:03.712112 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.712117 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.712121 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712126 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.712130 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712134 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.712139 | controller | -- \' 2026-03-09 19:10:03.712143 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712161 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.712165 | controller | - ' logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.712169 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.712176 | controller | - ' logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.712180 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.712184 | controller | found' 2026-03-09 19:10:03.712189 | controller | - ' logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.712193 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.712197 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.712201 | controller | - ' ' 2026-03-09 19:10:03.712206 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.712210 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.712214 | controller | - ' ' 2026-03-09 19:10:03.712219 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.712223 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.712227 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.712231 | controller | - ' exit 1' 2026-03-09 19:10:03.712236 | controller | - ' fi' 2026-03-09 19:10:03.712240 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.712244 | controller | - ' ' 2026-03-09 19:10:03.712252 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.712258 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.712262 | controller | | base64 -d)' 2026-03-09 19:10:03.712266 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.712271 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.712275 | controller | - ' exit 1' 2026-03-09 19:10:03.712279 | controller | - ' fi' 2026-03-09 19:10:03.712283 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.712288 | controller | - ' ' 2026-03-09 19:10:03.712292 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.712296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.712301 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.712305 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.712309 | controller | --timeout=300s' 2026-03-09 19:10:03.712313 | controller | - ' ' 2026-03-09 19:10:03.712318 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.712322 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.712326 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.712330 | controller | - ' ' 2026-03-09 19:10:03.712335 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712339 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712343 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712348 | controller | - ' ' 2026-03-09 19:10:03.712352 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.712356 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.712360 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.712364 | controller | available"' 2026-03-09 19:10:03.712369 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.712375 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.712380 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.712384 | controller | - ' exit 1' 2026-03-09 19:10:03.712388 | controller | - ' fi' 2026-03-09 19:10:03.712392 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.712397 | controller | - ' ' 2026-03-09 19:10:03.712409 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.712414 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.712418 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712423 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.712428 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712433 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.712438 | controller | -- \' 2026-03-09 19:10:03.712442 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712446 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.712451 | controller | - ' logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.712455 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.712459 | controller | - ' logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.712463 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.712468 | controller | found' 2026-03-09 19:10:03.712472 | controller | - ' logger.go:42: 18:53:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.712476 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.712481 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.712485 | controller | - ' ' 2026-03-09 19:10:03.712489 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.712493 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.712498 | controller | - ' ' 2026-03-09 19:10:03.712502 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.712506 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.712510 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.712515 | controller | - ' exit 1' 2026-03-09 19:10:03.712519 | controller | - ' fi' 2026-03-09 19:10:03.712523 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.712528 | controller | - ' ' 2026-03-09 19:10:03.712532 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.712536 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.712540 | controller | | base64 -d)' 2026-03-09 19:10:03.712545 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.712549 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.712553 | controller | - ' exit 1' 2026-03-09 19:10:03.712557 | controller | - ' fi' 2026-03-09 19:10:03.712562 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.712566 | controller | - ' ' 2026-03-09 19:10:03.712570 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.712574 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.712581 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.712587 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.712591 | controller | --timeout=300s' 2026-03-09 19:10:03.712595 | controller | - ' ' 2026-03-09 19:10:03.712600 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.712604 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.712608 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.712612 | controller | - ' ' 2026-03-09 19:10:03.712617 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712621 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712640 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712644 | controller | - ' ' 2026-03-09 19:10:03.712648 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.712653 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.712657 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.712661 | controller | available"' 2026-03-09 19:10:03.712665 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.712670 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.712674 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.712678 | controller | - ' exit 1' 2026-03-09 19:10:03.712683 | controller | - ' fi' 2026-03-09 19:10:03.712687 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.712691 | controller | - ' ' 2026-03-09 19:10:03.712695 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.712706 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.712711 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712715 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.712719 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712724 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.712728 | controller | -- \' 2026-03-09 19:10:03.712732 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712737 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.712741 | controller | - ' logger.go:42: 18:53:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.712745 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.712749 | controller | - ' logger.go:42: 18:53:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.712754 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.712758 | controller | found' 2026-03-09 19:10:03.712762 | controller | - ' logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.712766 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.712771 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.712775 | controller | - ' ' 2026-03-09 19:10:03.712779 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.712786 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.712790 | controller | - ' ' 2026-03-09 19:10:03.712794 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.712798 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.712803 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.712807 | controller | - ' exit 1' 2026-03-09 19:10:03.712811 | controller | - ' fi' 2026-03-09 19:10:03.712816 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.712820 | controller | - ' ' 2026-03-09 19:10:03.712824 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.712828 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.712833 | controller | | base64 -d)' 2026-03-09 19:10:03.712837 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.712841 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.712845 | controller | - ' exit 1' 2026-03-09 19:10:03.712850 | controller | - ' fi' 2026-03-09 19:10:03.712854 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.712858 | controller | - ' ' 2026-03-09 19:10:03.712862 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.712867 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.712871 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.712875 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.712879 | controller | --timeout=300s' 2026-03-09 19:10:03.712884 | controller | - ' ' 2026-03-09 19:10:03.712888 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.712893 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.712897 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.712902 | controller | - ' ' 2026-03-09 19:10:03.712906 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712910 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712915 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.712919 | controller | - ' ' 2026-03-09 19:10:03.712925 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.712929 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.712934 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.712938 | controller | available"' 2026-03-09 19:10:03.712942 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.712947 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.712951 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.712955 | controller | - ' exit 1' 2026-03-09 19:10:03.712960 | controller | - ' fi' 2026-03-09 19:10:03.712964 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.712968 | controller | - ' ' 2026-03-09 19:10:03.712972 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.712985 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.712990 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.712994 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.713006 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713011 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.713015 | controller | -- \' 2026-03-09 19:10:03.713019 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713024 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.713028 | controller | - ' logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.713032 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.713037 | controller | - ' logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.713041 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.713045 | controller | found' 2026-03-09 19:10:03.713049 | controller | - ' logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.713053 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.713058 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.713062 | controller | - ' ' 2026-03-09 19:10:03.713066 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.713071 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.713075 | controller | - ' ' 2026-03-09 19:10:03.713079 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.713083 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.713088 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.713092 | controller | - ' exit 1' 2026-03-09 19:10:03.713096 | controller | - ' fi' 2026-03-09 19:10:03.713100 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.713105 | controller | - ' ' 2026-03-09 19:10:03.713109 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.713113 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.713117 | controller | | base64 -d)' 2026-03-09 19:10:03.713122 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.713126 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.713130 | controller | - ' exit 1' 2026-03-09 19:10:03.713135 | controller | - ' fi' 2026-03-09 19:10:03.713139 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.713143 | controller | - ' ' 2026-03-09 19:10:03.713218 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.713223 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.713228 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.713232 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.713236 | controller | --timeout=300s' 2026-03-09 19:10:03.713240 | controller | - ' ' 2026-03-09 19:10:03.713245 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.713249 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.713255 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.713260 | controller | - ' ' 2026-03-09 19:10:03.713266 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713271 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713275 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713279 | controller | - ' ' 2026-03-09 19:10:03.713285 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.713289 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.713294 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.713298 | controller | available"' 2026-03-09 19:10:03.713302 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.713306 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.713311 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.713315 | controller | - ' exit 1' 2026-03-09 19:10:03.713319 | controller | - ' fi' 2026-03-09 19:10:03.713323 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.713328 | controller | - ' ' 2026-03-09 19:10:03.713332 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.713336 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.713349 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713353 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.713358 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713362 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.713366 | controller | -- \' 2026-03-09 19:10:03.713371 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713375 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.713379 | controller | - ' logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.713384 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.713388 | controller | - ' logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.713392 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.713397 | controller | found' 2026-03-09 19:10:03.713401 | controller | - ' logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.713405 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.713410 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.713414 | controller | - ' ' 2026-03-09 19:10:03.713418 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.713422 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.713427 | controller | - ' ' 2026-03-09 19:10:03.713431 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.713435 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.713440 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.713444 | controller | - ' exit 1' 2026-03-09 19:10:03.713450 | controller | - ' fi' 2026-03-09 19:10:03.713454 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.713459 | controller | - ' ' 2026-03-09 19:10:03.713463 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.713467 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.713472 | controller | | base64 -d)' 2026-03-09 19:10:03.713476 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.713480 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.713485 | controller | - ' exit 1' 2026-03-09 19:10:03.713489 | controller | - ' fi' 2026-03-09 19:10:03.713493 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.713498 | controller | - ' ' 2026-03-09 19:10:03.713502 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.713506 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.713511 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.713515 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.713519 | controller | --timeout=300s' 2026-03-09 19:10:03.713523 | controller | - ' ' 2026-03-09 19:10:03.713528 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.713532 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.713536 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.713540 | controller | - ' ' 2026-03-09 19:10:03.713545 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713549 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713553 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713558 | controller | - ' ' 2026-03-09 19:10:03.713562 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.713566 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.713571 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.713575 | controller | available"' 2026-03-09 19:10:03.713579 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.713588 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.713592 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.713596 | controller | - ' exit 1' 2026-03-09 19:10:03.713601 | controller | - ' fi' 2026-03-09 19:10:03.713605 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.713609 | controller | - ' ' 2026-03-09 19:10:03.713614 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.713618 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.713632 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713637 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.713652 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713656 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.713662 | controller | -- \' 2026-03-09 19:10:03.713667 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713671 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.713675 | controller | - ' logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.713680 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.713684 | controller | - ' logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.713688 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.713692 | controller | found' 2026-03-09 19:10:03.713697 | controller | - ' logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.713701 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.713705 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.713710 | controller | - ' ' 2026-03-09 19:10:03.713714 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.713718 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.713723 | controller | - ' ' 2026-03-09 19:10:03.713727 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.713731 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.713762 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.713767 | controller | - ' exit 1' 2026-03-09 19:10:03.713771 | controller | - ' fi' 2026-03-09 19:10:03.713776 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.713782 | controller | - ' ' 2026-03-09 19:10:03.713787 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.713792 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.713796 | controller | | base64 -d)' 2026-03-09 19:10:03.713801 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.713805 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.713809 | controller | - ' exit 1' 2026-03-09 19:10:03.713814 | controller | - ' fi' 2026-03-09 19:10:03.713818 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.713822 | controller | - ' ' 2026-03-09 19:10:03.713826 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.713831 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.713835 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.713839 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.713844 | controller | --timeout=300s' 2026-03-09 19:10:03.713848 | controller | - ' ' 2026-03-09 19:10:03.713852 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.713856 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.713861 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.713865 | controller | - ' ' 2026-03-09 19:10:03.713869 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713874 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713878 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.713884 | controller | - ' ' 2026-03-09 19:10:03.713889 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.713893 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.713899 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.713904 | controller | available"' 2026-03-09 19:10:03.713908 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.713913 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.713918 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.713923 | controller | - ' exit 1' 2026-03-09 19:10:03.713928 | controller | - ' fi' 2026-03-09 19:10:03.713932 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.713936 | controller | - ' ' 2026-03-09 19:10:03.713941 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.713945 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.713949 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713961 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.713965 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713970 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.713974 | controller | -- \' 2026-03-09 19:10:03.713978 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.713983 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.713988 | controller | - ' logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.713993 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.714001 | controller | - ' logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.714005 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.714009 | controller | found' 2026-03-09 19:10:03.714014 | controller | - ' logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.714018 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.714022 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.714027 | controller | - ' ' 2026-03-09 19:10:03.714031 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.714035 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.714039 | controller | - ' ' 2026-03-09 19:10:03.714044 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.714050 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.714055 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.714059 | controller | - ' exit 1' 2026-03-09 19:10:03.714063 | controller | - ' fi' 2026-03-09 19:10:03.714067 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.714071 | controller | - ' ' 2026-03-09 19:10:03.714076 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.714080 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.714084 | controller | | base64 -d)' 2026-03-09 19:10:03.714088 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.714093 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.714099 | controller | - ' exit 1' 2026-03-09 19:10:03.714104 | controller | - ' fi' 2026-03-09 19:10:03.714112 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.714116 | controller | - ' ' 2026-03-09 19:10:03.714120 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.714125 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.714129 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.714133 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.714137 | controller | --timeout=300s' 2026-03-09 19:10:03.714142 | controller | - ' ' 2026-03-09 19:10:03.714164 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.714171 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.714175 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.714179 | controller | - ' ' 2026-03-09 19:10:03.714184 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714190 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714194 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714198 | controller | - ' ' 2026-03-09 19:10:03.714203 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.714207 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.714211 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.714215 | controller | available"' 2026-03-09 19:10:03.714220 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.714224 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.714228 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.714232 | controller | - ' exit 1' 2026-03-09 19:10:03.714237 | controller | - ' fi' 2026-03-09 19:10:03.714241 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.714245 | controller | - ' ' 2026-03-09 19:10:03.714249 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.714254 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.714268 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714273 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.714285 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714289 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.714294 | controller | -- \' 2026-03-09 19:10:03.714298 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714302 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.714307 | controller | - ' logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.714311 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.714315 | controller | - ' logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.714322 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.714326 | controller | found' 2026-03-09 19:10:03.714330 | controller | - ' logger.go:42: 18:53:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.714335 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.714339 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.714343 | controller | - ' ' 2026-03-09 19:10:03.714347 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.714352 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.714356 | controller | - ' ' 2026-03-09 19:10:03.714360 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.714364 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.714369 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.714373 | controller | - ' exit 1' 2026-03-09 19:10:03.714377 | controller | - ' fi' 2026-03-09 19:10:03.714381 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.714386 | controller | - ' ' 2026-03-09 19:10:03.714390 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.714394 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.714398 | controller | | base64 -d)' 2026-03-09 19:10:03.714403 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.714407 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.714411 | controller | - ' exit 1' 2026-03-09 19:10:03.714415 | controller | - ' fi' 2026-03-09 19:10:03.714420 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.714424 | controller | - ' ' 2026-03-09 19:10:03.714428 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.714433 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.714437 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.714441 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.714445 | controller | --timeout=300s' 2026-03-09 19:10:03.714450 | controller | - ' ' 2026-03-09 19:10:03.714454 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.714458 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.714462 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.714467 | controller | - ' ' 2026-03-09 19:10:03.714471 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714475 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714479 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714484 | controller | - ' ' 2026-03-09 19:10:03.714488 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.714492 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.714496 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.714501 | controller | available"' 2026-03-09 19:10:03.714505 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.714512 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.714518 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.714522 | controller | - ' exit 1' 2026-03-09 19:10:03.714527 | controller | - ' fi' 2026-03-09 19:10:03.714531 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.714535 | controller | - ' ' 2026-03-09 19:10:03.714539 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.714544 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.714548 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714552 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.714563 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714568 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.714572 | controller | -- \' 2026-03-09 19:10:03.714577 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714581 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.714585 | controller | - ' logger.go:42: 18:53:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.714589 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.714594 | controller | - ' logger.go:42: 18:53:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.714598 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.714602 | controller | found' 2026-03-09 19:10:03.714607 | controller | - ' logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.714611 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.714615 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.714619 | controller | - ' ' 2026-03-09 19:10:03.714624 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.714628 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.714632 | controller | - ' ' 2026-03-09 19:10:03.714636 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.714641 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.714645 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.714649 | controller | - ' exit 1' 2026-03-09 19:10:03.714654 | controller | - ' fi' 2026-03-09 19:10:03.714658 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.714662 | controller | - ' ' 2026-03-09 19:10:03.714666 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.714671 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.714675 | controller | | base64 -d)' 2026-03-09 19:10:03.714679 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.714685 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.714690 | controller | - ' exit 1' 2026-03-09 19:10:03.714694 | controller | - ' fi' 2026-03-09 19:10:03.714698 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.714702 | controller | - ' ' 2026-03-09 19:10:03.714707 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.714711 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.714715 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.714721 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.714725 | controller | --timeout=300s' 2026-03-09 19:10:03.714730 | controller | - ' ' 2026-03-09 19:10:03.714734 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.714738 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.714743 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.714747 | controller | - ' ' 2026-03-09 19:10:03.714751 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714755 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714759 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.714764 | controller | - ' ' 2026-03-09 19:10:03.714768 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.714772 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.714776 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.714781 | controller | available"' 2026-03-09 19:10:03.714785 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.714789 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.714793 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.714798 | controller | - ' exit 1' 2026-03-09 19:10:03.714802 | controller | - ' fi' 2026-03-09 19:10:03.714806 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.714810 | controller | - ' ' 2026-03-09 19:10:03.714815 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.714819 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.714823 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714827 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.714839 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714843 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.714848 | controller | -- \' 2026-03-09 19:10:03.714852 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.714856 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.714861 | controller | - ' logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.714865 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.714869 | controller | - ' logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.714874 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.714878 | controller | found' 2026-03-09 19:10:03.714882 | controller | - ' logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.714886 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.714890 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.714895 | controller | - ' ' 2026-03-09 19:10:03.714899 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.714906 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.714910 | controller | - ' ' 2026-03-09 19:10:03.714915 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.714919 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.714923 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.714928 | controller | - ' exit 1' 2026-03-09 19:10:03.714932 | controller | - ' fi' 2026-03-09 19:10:03.715128 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.715139 | controller | - ' ' 2026-03-09 19:10:03.715157 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.715164 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.715169 | controller | | base64 -d)' 2026-03-09 19:10:03.715173 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.715177 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.715181 | controller | - ' exit 1' 2026-03-09 19:10:03.715186 | controller | - ' fi' 2026-03-09 19:10:03.715190 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.715194 | controller | - ' ' 2026-03-09 19:10:03.715199 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.715203 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.715207 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.715211 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.715215 | controller | --timeout=300s' 2026-03-09 19:10:03.715220 | controller | - ' ' 2026-03-09 19:10:03.715224 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.715228 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.715233 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.715237 | controller | - ' ' 2026-03-09 19:10:03.715241 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715245 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715250 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715254 | controller | - ' ' 2026-03-09 19:10:03.715258 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.715262 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.715267 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.715271 | controller | available"' 2026-03-09 19:10:03.715275 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.715279 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.715284 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.715288 | controller | - ' exit 1' 2026-03-09 19:10:03.715292 | controller | - ' fi' 2026-03-09 19:10:03.715296 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.715301 | controller | - ' ' 2026-03-09 19:10:03.715308 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.715312 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.715319 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715323 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.715327 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715339 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.715346 | controller | -- \' 2026-03-09 19:10:03.715350 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715354 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.715359 | controller | - ' logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.715363 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.715367 | controller | - ' logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.715371 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.715376 | controller | found' 2026-03-09 19:10:03.715380 | controller | - ' logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.715384 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.715388 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.715393 | controller | - ' ' 2026-03-09 19:10:03.715397 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.715401 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.715405 | controller | - ' ' 2026-03-09 19:10:03.715410 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.715414 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.715418 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.715422 | controller | - ' exit 1' 2026-03-09 19:10:03.715428 | controller | - ' fi' 2026-03-09 19:10:03.715433 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.715437 | controller | - ' ' 2026-03-09 19:10:03.715441 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.715445 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.715450 | controller | | base64 -d)' 2026-03-09 19:10:03.715454 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.715458 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.715462 | controller | - ' exit 1' 2026-03-09 19:10:03.715467 | controller | - ' fi' 2026-03-09 19:10:03.715471 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.715475 | controller | - ' ' 2026-03-09 19:10:03.715479 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.715484 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.715488 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.715492 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.715496 | controller | --timeout=300s' 2026-03-09 19:10:03.715501 | controller | - ' ' 2026-03-09 19:10:03.715505 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.715509 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.715513 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.715520 | controller | - ' ' 2026-03-09 19:10:03.715524 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715528 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715532 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715537 | controller | - ' ' 2026-03-09 19:10:03.715541 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.715545 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.715549 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.715554 | controller | available"' 2026-03-09 19:10:03.715558 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.715562 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.715566 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.715571 | controller | - ' exit 1' 2026-03-09 19:10:03.715575 | controller | - ' fi' 2026-03-09 19:10:03.715579 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.715583 | controller | - ' ' 2026-03-09 19:10:03.715588 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.715592 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.715596 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715601 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.715605 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715617 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.715621 | controller | -- \' 2026-03-09 19:10:03.715626 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715630 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.715634 | controller | - ' logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.715638 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.715643 | controller | - ' logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.715647 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.715651 | controller | found' 2026-03-09 19:10:03.715656 | controller | - ' logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.715660 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.715664 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.715668 | controller | - ' ' 2026-03-09 19:10:03.715673 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.715677 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.715681 | controller | - ' ' 2026-03-09 19:10:03.715685 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.715690 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.715694 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.715698 | controller | - ' exit 1' 2026-03-09 19:10:03.715702 | controller | - ' fi' 2026-03-09 19:10:03.715707 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.715731 | controller | - ' ' 2026-03-09 19:10:03.715737 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.715741 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.715745 | controller | | base64 -d)' 2026-03-09 19:10:03.715750 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.715754 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.715767 | controller | - ' exit 1' 2026-03-09 19:10:03.715772 | controller | - ' fi' 2026-03-09 19:10:03.715776 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.715780 | controller | - ' ' 2026-03-09 19:10:03.715785 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.715789 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.715793 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.715797 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.715802 | controller | --timeout=300s' 2026-03-09 19:10:03.715813 | controller | - ' ' 2026-03-09 19:10:03.715817 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.715822 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.715826 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.715830 | controller | - ' ' 2026-03-09 19:10:03.715834 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715839 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715843 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.715847 | controller | - ' ' 2026-03-09 19:10:03.715851 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.715856 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.715860 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.715864 | controller | available"' 2026-03-09 19:10:03.715868 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.715873 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.715877 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.715881 | controller | - ' exit 1' 2026-03-09 19:10:03.715885 | controller | - ' fi' 2026-03-09 19:10:03.715890 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.715894 | controller | - ' ' 2026-03-09 19:10:03.715898 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.715902 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.715907 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715913 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.715917 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715922 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.715926 | controller | -- \' 2026-03-09 19:10:03.715941 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.715946 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.715950 | controller | - ' logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.715954 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.715959 | controller | - ' logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.715963 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.715967 | controller | found' 2026-03-09 19:10:03.715971 | controller | - ' logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.715976 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.715980 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.715984 | controller | - ' ' 2026-03-09 19:10:03.715988 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.715993 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.715997 | controller | - ' ' 2026-03-09 19:10:03.716001 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.716006 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.716010 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.716014 | controller | - ' exit 1' 2026-03-09 19:10:03.716018 | controller | - ' fi' 2026-03-09 19:10:03.716023 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.716027 | controller | - ' ' 2026-03-09 19:10:03.716031 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.716035 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.716040 | controller | | base64 -d)' 2026-03-09 19:10:03.716044 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.716048 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.716052 | controller | - ' exit 1' 2026-03-09 19:10:03.716057 | controller | - ' fi' 2026-03-09 19:10:03.716061 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.716065 | controller | - ' ' 2026-03-09 19:10:03.716069 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.716074 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.716078 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.716082 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.716086 | controller | --timeout=300s' 2026-03-09 19:10:03.716091 | controller | - ' ' 2026-03-09 19:10:03.716095 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.716099 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.716103 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.716108 | controller | - ' ' 2026-03-09 19:10:03.716112 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716116 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716120 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716125 | controller | - ' ' 2026-03-09 19:10:03.716131 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.716135 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.716139 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.716155 | controller | available"' 2026-03-09 19:10:03.716162 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.716166 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.716170 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.716175 | controller | - ' exit 1' 2026-03-09 19:10:03.716179 | controller | - ' fi' 2026-03-09 19:10:03.716183 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.716187 | controller | - ' ' 2026-03-09 19:10:03.716192 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.716196 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.716200 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716205 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.716209 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716213 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.716217 | controller | -- \' 2026-03-09 19:10:03.716229 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716234 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.716240 | controller | - ' logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.716245 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.716249 | controller | - ' logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.716253 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.716257 | controller | found' 2026-03-09 19:10:03.716262 | controller | - ' logger.go:42: 18:53:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.716266 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.716270 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.716275 | controller | - ' ' 2026-03-09 19:10:03.716279 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.716283 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.716289 | controller | - ' ' 2026-03-09 19:10:03.716294 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.716298 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.716303 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.716307 | controller | - ' exit 1' 2026-03-09 19:10:03.716311 | controller | - ' fi' 2026-03-09 19:10:03.716316 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.716320 | controller | - ' ' 2026-03-09 19:10:03.716324 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.716328 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.716333 | controller | | base64 -d)' 2026-03-09 19:10:03.716337 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.716341 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.716348 | controller | - ' exit 1' 2026-03-09 19:10:03.716352 | controller | - ' fi' 2026-03-09 19:10:03.716356 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.716361 | controller | - ' ' 2026-03-09 19:10:03.716365 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.716369 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.716373 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.716378 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.716382 | controller | --timeout=300s' 2026-03-09 19:10:03.716388 | controller | - ' ' 2026-03-09 19:10:03.716392 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.716397 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.716401 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.716405 | controller | - ' ' 2026-03-09 19:10:03.716410 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716414 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716418 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716422 | controller | - ' ' 2026-03-09 19:10:03.716427 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.716431 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.716435 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.716439 | controller | available"' 2026-03-09 19:10:03.716444 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.716450 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.716454 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.716458 | controller | - ' exit 1' 2026-03-09 19:10:03.716466 | controller | - ' fi' 2026-03-09 19:10:03.716471 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.716475 | controller | - ' ' 2026-03-09 19:10:03.716479 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.716483 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.716488 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716492 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.716496 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716501 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.716505 | controller | -- \' 2026-03-09 19:10:03.716509 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716521 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.716526 | controller | - ' logger.go:42: 18:53:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.716530 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.716536 | controller | - ' logger.go:42: 18:53:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.716542 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.716546 | controller | found' 2026-03-09 19:10:03.716551 | controller | - ' logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.716555 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.716559 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.716564 | controller | - ' ' 2026-03-09 19:10:03.716568 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.716572 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.716576 | controller | - ' ' 2026-03-09 19:10:03.716581 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.716585 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.716589 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.716593 | controller | - ' exit 1' 2026-03-09 19:10:03.716598 | controller | - ' fi' 2026-03-09 19:10:03.716602 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.716606 | controller | - ' ' 2026-03-09 19:10:03.716611 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.716615 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.716619 | controller | | base64 -d)' 2026-03-09 19:10:03.716623 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.716628 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.716632 | controller | - ' exit 1' 2026-03-09 19:10:03.716636 | controller | - ' fi' 2026-03-09 19:10:03.716640 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.716645 | controller | - ' ' 2026-03-09 19:10:03.716649 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.716653 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.716658 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.716662 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.716666 | controller | --timeout=300s' 2026-03-09 19:10:03.716670 | controller | - ' ' 2026-03-09 19:10:03.716675 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.716679 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.716683 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.716687 | controller | - ' ' 2026-03-09 19:10:03.716692 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716696 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716700 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716705 | controller | - ' ' 2026-03-09 19:10:03.716709 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.716713 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.716717 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.716722 | controller | available"' 2026-03-09 19:10:03.716726 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.716730 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.716734 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.716741 | controller | - ' exit 1' 2026-03-09 19:10:03.716745 | controller | - ' fi' 2026-03-09 19:10:03.716749 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.716754 | controller | - ' ' 2026-03-09 19:10:03.716758 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.716762 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.716766 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716771 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.716775 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716779 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.716783 | controller | -- \' 2026-03-09 19:10:03.716788 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.716799 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.716803 | controller | - ' logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.716808 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.716812 | controller | - ' logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.716816 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.716821 | controller | found' 2026-03-09 19:10:03.716825 | controller | - ' logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.716829 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.716833 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.716838 | controller | - ' ' 2026-03-09 19:10:03.716842 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.716846 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.716850 | controller | - ' ' 2026-03-09 19:10:03.716855 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.716859 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.716863 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.716867 | controller | - ' exit 1' 2026-03-09 19:10:03.716872 | controller | - ' fi' 2026-03-09 19:10:03.716876 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.716880 | controller | - ' ' 2026-03-09 19:10:03.716884 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.716889 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.716893 | controller | | base64 -d)' 2026-03-09 19:10:03.716897 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.716901 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.716906 | controller | - ' exit 1' 2026-03-09 19:10:03.716910 | controller | - ' fi' 2026-03-09 19:10:03.716914 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.716918 | controller | - ' ' 2026-03-09 19:10:03.716923 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.716927 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.716931 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.716935 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.716942 | controller | --timeout=300s' 2026-03-09 19:10:03.716946 | controller | - ' ' 2026-03-09 19:10:03.716950 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.716955 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.716959 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.716963 | controller | - ' ' 2026-03-09 19:10:03.716967 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716972 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716976 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.716980 | controller | - ' ' 2026-03-09 19:10:03.716984 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.716989 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.716993 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.716997 | controller | available"' 2026-03-09 19:10:03.717001 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.717006 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.717010 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.717014 | controller | - ' exit 1' 2026-03-09 19:10:03.717018 | controller | - ' fi' 2026-03-09 19:10:03.717023 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.717027 | controller | - ' ' 2026-03-09 19:10:03.717031 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.717035 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.717040 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717044 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.717048 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717052 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.717057 | controller | -- \' 2026-03-09 19:10:03.717061 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717065 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.717078 | controller | - ' logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.717082 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.717087 | controller | - ' logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.717091 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.717095 | controller | found' 2026-03-09 19:10:03.717100 | controller | - ' logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.717104 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.717108 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.717113 | controller | - ' ' 2026-03-09 19:10:03.717117 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.717121 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.717127 | controller | - ' ' 2026-03-09 19:10:03.717131 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.717136 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.717140 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.717154 | controller | - ' exit 1' 2026-03-09 19:10:03.717161 | controller | - ' fi' 2026-03-09 19:10:03.717165 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.717169 | controller | - ' ' 2026-03-09 19:10:03.717174 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.717178 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.717182 | controller | | base64 -d)' 2026-03-09 19:10:03.717186 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.717191 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.717195 | controller | - ' exit 1' 2026-03-09 19:10:03.717199 | controller | - ' fi' 2026-03-09 19:10:03.717203 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.717208 | controller | - ' ' 2026-03-09 19:10:03.717212 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.717216 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.717221 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.717225 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.717229 | controller | --timeout=300s' 2026-03-09 19:10:03.717233 | controller | - ' ' 2026-03-09 19:10:03.717238 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.717242 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.717246 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.717250 | controller | - ' ' 2026-03-09 19:10:03.717255 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717259 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717263 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717267 | controller | - ' ' 2026-03-09 19:10:03.717272 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.717276 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.717280 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.717284 | controller | available"' 2026-03-09 19:10:03.717289 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.717293 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.717297 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.717302 | controller | - ' exit 1' 2026-03-09 19:10:03.717306 | controller | - ' fi' 2026-03-09 19:10:03.717310 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.717314 | controller | - ' ' 2026-03-09 19:10:03.717319 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.717323 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.717327 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717333 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.717337 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717342 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.717346 | controller | -- \' 2026-03-09 19:10:03.717350 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717355 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.717369 | controller | - ' logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.717373 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.717378 | controller | - ' logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.717382 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.717387 | controller | found' 2026-03-09 19:10:03.717391 | controller | - ' logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.717395 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.717399 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.717404 | controller | - ' ' 2026-03-09 19:10:03.717408 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.717412 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.717416 | controller | - ' ' 2026-03-09 19:10:03.717421 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.717425 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.717429 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.717433 | controller | - ' exit 1' 2026-03-09 19:10:03.717438 | controller | - ' fi' 2026-03-09 19:10:03.717442 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.717446 | controller | - ' ' 2026-03-09 19:10:03.717450 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.717455 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.717459 | controller | | base64 -d)' 2026-03-09 19:10:03.717463 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.717467 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.717472 | controller | - ' exit 1' 2026-03-09 19:10:03.717476 | controller | - ' fi' 2026-03-09 19:10:03.717480 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.717484 | controller | - ' ' 2026-03-09 19:10:03.717489 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.717493 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.717497 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.717502 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.717506 | controller | --timeout=300s' 2026-03-09 19:10:03.717510 | controller | - ' ' 2026-03-09 19:10:03.717514 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.717520 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.717525 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.717531 | controller | - ' ' 2026-03-09 19:10:03.717536 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717541 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717545 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717549 | controller | - ' ' 2026-03-09 19:10:03.717553 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.717558 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.717562 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.717566 | controller | available"' 2026-03-09 19:10:03.717570 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.717575 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.717579 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.717583 | controller | - ' exit 1' 2026-03-09 19:10:03.717587 | controller | - ' fi' 2026-03-09 19:10:03.717592 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.717596 | controller | - ' ' 2026-03-09 19:10:03.717600 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.717604 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.717608 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717613 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.717617 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717621 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.717626 | controller | -- \' 2026-03-09 19:10:03.717630 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717634 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.717638 | controller | - ' logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.717649 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.717653 | controller | - ' logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.717659 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.717664 | controller | found' 2026-03-09 19:10:03.717668 | controller | - ' logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.717672 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.717676 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.717682 | controller | - ' ' 2026-03-09 19:10:03.717687 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.717691 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.717695 | controller | - ' ' 2026-03-09 19:10:03.717700 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.717704 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.717708 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.717713 | controller | - ' exit 1' 2026-03-09 19:10:03.717717 | controller | - ' fi' 2026-03-09 19:10:03.717721 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.717727 | controller | - ' ' 2026-03-09 19:10:03.717735 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.717739 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.717743 | controller | | base64 -d)' 2026-03-09 19:10:03.717748 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.717752 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.717756 | controller | - ' exit 1' 2026-03-09 19:10:03.717761 | controller | - ' fi' 2026-03-09 19:10:03.717765 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.717769 | controller | - ' ' 2026-03-09 19:10:03.717773 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.717778 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.717782 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.717786 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.717790 | controller | --timeout=300s' 2026-03-09 19:10:03.717795 | controller | - ' ' 2026-03-09 19:10:03.717799 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.717803 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.717808 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.717812 | controller | - ' ' 2026-03-09 19:10:03.717816 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717821 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717825 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.717829 | controller | - ' ' 2026-03-09 19:10:03.717833 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.717838 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.717842 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.717846 | controller | available"' 2026-03-09 19:10:03.717851 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.717855 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.717859 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.717863 | controller | - ' exit 1' 2026-03-09 19:10:03.717868 | controller | - ' fi' 2026-03-09 19:10:03.717872 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.717876 | controller | - ' ' 2026-03-09 19:10:03.717881 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.717885 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.717889 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717894 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.717898 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717902 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.717906 | controller | -- \' 2026-03-09 19:10:03.717911 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.717917 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.717921 | controller | - ' logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.717925 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.717936 | controller | - ' logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.717941 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.717945 | controller | found' 2026-03-09 19:10:03.717949 | controller | - ' logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.717954 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.717958 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.717962 | controller | - ' ' 2026-03-09 19:10:03.717967 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.717971 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.717975 | controller | - ' ' 2026-03-09 19:10:03.717980 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.717984 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.717988 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.717992 | controller | - ' exit 1' 2026-03-09 19:10:03.717997 | controller | - ' fi' 2026-03-09 19:10:03.718001 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.718005 | controller | - ' ' 2026-03-09 19:10:03.718009 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.718014 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.718018 | controller | | base64 -d)' 2026-03-09 19:10:03.718022 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.718026 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.718031 | controller | - ' exit 1' 2026-03-09 19:10:03.718035 | controller | - ' fi' 2026-03-09 19:10:03.718039 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.718043 | controller | - ' ' 2026-03-09 19:10:03.718048 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.718052 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.718056 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.718060 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.718065 | controller | --timeout=300s' 2026-03-09 19:10:03.718069 | controller | - ' ' 2026-03-09 19:10:03.718073 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.718077 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.718082 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.718086 | controller | - ' ' 2026-03-09 19:10:03.718090 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718095 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718099 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718103 | controller | - ' ' 2026-03-09 19:10:03.718107 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.718113 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.718117 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.718122 | controller | available"' 2026-03-09 19:10:03.718126 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.718130 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.718134 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.718139 | controller | - ' exit 1' 2026-03-09 19:10:03.718143 | controller | - ' fi' 2026-03-09 19:10:03.718171 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.718175 | controller | - ' ' 2026-03-09 19:10:03.718180 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.718184 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.718188 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718192 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.718197 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718201 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.718205 | controller | -- \' 2026-03-09 19:10:03.718210 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718214 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.718218 | controller | - ' logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.718222 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.718227 | controller | - ' logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.718239 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.718243 | controller | found' 2026-03-09 19:10:03.718250 | controller | - ' logger.go:42: 18:53:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.718255 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.718259 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.718263 | controller | - ' ' 2026-03-09 19:10:03.718268 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.718272 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.718276 | controller | - ' ' 2026-03-09 19:10:03.718280 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.718285 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.718289 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.718293 | controller | - ' exit 1' 2026-03-09 19:10:03.718297 | controller | - ' fi' 2026-03-09 19:10:03.718302 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.718306 | controller | - ' ' 2026-03-09 19:10:03.718310 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.718314 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.718319 | controller | | base64 -d)' 2026-03-09 19:10:03.718323 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.718327 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.718331 | controller | - ' exit 1' 2026-03-09 19:10:03.718336 | controller | - ' fi' 2026-03-09 19:10:03.718342 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.718346 | controller | - ' ' 2026-03-09 19:10:03.718351 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.718355 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.718359 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.718363 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.718368 | controller | --timeout=300s' 2026-03-09 19:10:03.718372 | controller | - ' ' 2026-03-09 19:10:03.718376 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.718380 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.718385 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.718389 | controller | - ' ' 2026-03-09 19:10:03.718393 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718397 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718402 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718406 | controller | - ' ' 2026-03-09 19:10:03.718410 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.718416 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.718420 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.718425 | controller | available"' 2026-03-09 19:10:03.718429 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.718433 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.718437 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.718442 | controller | - ' exit 1' 2026-03-09 19:10:03.718446 | controller | - ' fi' 2026-03-09 19:10:03.718450 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.718454 | controller | - ' ' 2026-03-09 19:10:03.718459 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.718463 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.718467 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718471 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.718476 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718480 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.718484 | controller | -- \' 2026-03-09 19:10:03.718488 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718493 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.718497 | controller | - ' logger.go:42: 18:53:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.718501 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.718505 | controller | - ' logger.go:42: 18:53:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.718517 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.718523 | controller | found' 2026-03-09 19:10:03.718528 | controller | - ' logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.718532 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.718536 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.718541 | controller | - ' ' 2026-03-09 19:10:03.718545 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.718549 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.718553 | controller | - ' ' 2026-03-09 19:10:03.718558 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.718562 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.718566 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.718570 | controller | - ' exit 1' 2026-03-09 19:10:03.718575 | controller | - ' fi' 2026-03-09 19:10:03.718581 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.718585 | controller | - ' ' 2026-03-09 19:10:03.718591 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.718595 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.718599 | controller | | base64 -d)' 2026-03-09 19:10:03.718604 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.718608 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.718612 | controller | - ' exit 1' 2026-03-09 19:10:03.718616 | controller | - ' fi' 2026-03-09 19:10:03.718621 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.718625 | controller | - ' ' 2026-03-09 19:10:03.718629 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.718634 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.718638 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.718642 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.718646 | controller | --timeout=300s' 2026-03-09 19:10:03.718651 | controller | - ' ' 2026-03-09 19:10:03.718655 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.718659 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.718663 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.718668 | controller | - ' ' 2026-03-09 19:10:03.718672 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718676 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718680 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718685 | controller | - ' ' 2026-03-09 19:10:03.718689 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.718693 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.718698 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.718702 | controller | available"' 2026-03-09 19:10:03.718706 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.718710 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.718715 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.718721 | controller | - ' exit 1' 2026-03-09 19:10:03.718725 | controller | - ' fi' 2026-03-09 19:10:03.718729 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.718734 | controller | - ' ' 2026-03-09 19:10:03.718738 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.718742 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.718746 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718751 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.718755 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718759 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.718764 | controller | -- \' 2026-03-09 19:10:03.718768 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.718772 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.718776 | controller | - ' logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.718781 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.718785 | controller | - ' logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.718789 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.718794 | controller | found' 2026-03-09 19:10:03.718805 | controller | - ' logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.718810 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.718814 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.718818 | controller | - ' ' 2026-03-09 19:10:03.718824 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.718829 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.718833 | controller | - ' ' 2026-03-09 19:10:03.718837 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.718841 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.718846 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.718850 | controller | - ' exit 1' 2026-03-09 19:10:03.718854 | controller | - ' fi' 2026-03-09 19:10:03.718858 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.718863 | controller | - ' ' 2026-03-09 19:10:03.718867 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.718871 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.718875 | controller | | base64 -d)' 2026-03-09 19:10:03.718880 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.718886 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.718890 | controller | - ' exit 1' 2026-03-09 19:10:03.718894 | controller | - ' fi' 2026-03-09 19:10:03.718898 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.718903 | controller | - ' ' 2026-03-09 19:10:03.718907 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.718911 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.718915 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.718920 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.718926 | controller | --timeout=300s' 2026-03-09 19:10:03.718930 | controller | - ' ' 2026-03-09 19:10:03.718934 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.718939 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.718943 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.718947 | controller | - ' ' 2026-03-09 19:10:03.718951 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718956 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718960 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.718964 | controller | - ' ' 2026-03-09 19:10:03.718969 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.718973 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.718977 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.718981 | controller | available"' 2026-03-09 19:10:03.718986 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.718991 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.718996 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.719000 | controller | - ' exit 1' 2026-03-09 19:10:03.719006 | controller | - ' fi' 2026-03-09 19:10:03.719012 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.719018 | controller | - ' ' 2026-03-09 19:10:03.719024 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.719029 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.719035 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719041 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.719047 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719053 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.719057 | controller | -- \' 2026-03-09 19:10:03.719062 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719066 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.719070 | controller | - ' logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.719074 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.719079 | controller | - ' logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.719083 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.719087 | controller | found' 2026-03-09 19:10:03.719101 | controller | - ' logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.719107 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.719111 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.719115 | controller | - ' ' 2026-03-09 19:10:03.719120 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.719124 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.719130 | controller | - ' ' 2026-03-09 19:10:03.719135 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.719139 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.719143 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.719162 | controller | - ' exit 1' 2026-03-09 19:10:03.719166 | controller | - ' fi' 2026-03-09 19:10:03.719170 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.719175 | controller | - ' ' 2026-03-09 19:10:03.719179 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.719183 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.719187 | controller | | base64 -d)' 2026-03-09 19:10:03.719192 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.719196 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.719200 | controller | - ' exit 1' 2026-03-09 19:10:03.719204 | controller | - ' fi' 2026-03-09 19:10:03.719209 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.719213 | controller | - ' ' 2026-03-09 19:10:03.719217 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.719222 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.719226 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.719230 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.719234 | controller | --timeout=300s' 2026-03-09 19:10:03.719239 | controller | - ' ' 2026-03-09 19:10:03.719243 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.719247 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.719251 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.719256 | controller | - ' ' 2026-03-09 19:10:03.719260 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719264 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719268 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719273 | controller | - ' ' 2026-03-09 19:10:03.719277 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.719281 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.719285 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.719290 | controller | available"' 2026-03-09 19:10:03.719294 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.719298 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.719302 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.719307 | controller | - ' exit 1' 2026-03-09 19:10:03.719311 | controller | - ' fi' 2026-03-09 19:10:03.719315 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.719319 | controller | - ' ' 2026-03-09 19:10:03.719324 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.719328 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.719332 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719340 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.719344 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719348 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.719353 | controller | -- \' 2026-03-09 19:10:03.719357 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719361 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.719365 | controller | - ' logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.719370 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.719374 | controller | - ' logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.719378 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.719383 | controller | found' 2026-03-09 19:10:03.719387 | controller | - ' logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.719391 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.719403 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.719407 | controller | - ' ' 2026-03-09 19:10:03.719412 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.719419 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.719424 | controller | - ' ' 2026-03-09 19:10:03.719428 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.719433 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.719437 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.719441 | controller | - ' exit 1' 2026-03-09 19:10:03.719445 | controller | - ' fi' 2026-03-09 19:10:03.719450 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.719454 | controller | - ' ' 2026-03-09 19:10:03.719458 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.719463 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.719467 | controller | | base64 -d)' 2026-03-09 19:10:03.719471 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.719476 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.719480 | controller | - ' exit 1' 2026-03-09 19:10:03.719484 | controller | - ' fi' 2026-03-09 19:10:03.719488 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.719493 | controller | - ' ' 2026-03-09 19:10:03.719497 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.719501 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.719505 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.719510 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.719514 | controller | --timeout=300s' 2026-03-09 19:10:03.719518 | controller | - ' ' 2026-03-09 19:10:03.719522 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.719527 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.719533 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.719537 | controller | - ' ' 2026-03-09 19:10:03.719543 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719547 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719552 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719556 | controller | - ' ' 2026-03-09 19:10:03.719560 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.719564 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.719569 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.719573 | controller | available"' 2026-03-09 19:10:03.719577 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.719581 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.719586 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.719590 | controller | - ' exit 1' 2026-03-09 19:10:03.719594 | controller | - ' fi' 2026-03-09 19:10:03.719599 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.719603 | controller | - ' ' 2026-03-09 19:10:03.719609 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.719614 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.719618 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719622 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.719626 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719632 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.719638 | controller | -- \' 2026-03-09 19:10:03.719642 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719646 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.719651 | controller | - ' logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.719655 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.719659 | controller | - ' logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.719664 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.719668 | controller | found' 2026-03-09 19:10:03.719672 | controller | - ' logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.719676 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.719681 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.719685 | controller | - ' ' 2026-03-09 19:10:03.719696 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.719701 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.719724 | controller | - ' ' 2026-03-09 19:10:03.719729 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.719733 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.719738 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.719742 | controller | - ' exit 1' 2026-03-09 19:10:03.719746 | controller | - ' fi' 2026-03-09 19:10:03.719750 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.719755 | controller | - ' ' 2026-03-09 19:10:03.719759 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.719766 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.719770 | controller | | base64 -d)' 2026-03-09 19:10:03.719774 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.719779 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.719783 | controller | - ' exit 1' 2026-03-09 19:10:03.719787 | controller | - ' fi' 2026-03-09 19:10:03.719791 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.719796 | controller | - ' ' 2026-03-09 19:10:03.719800 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.719804 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.719808 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.719813 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.719817 | controller | --timeout=300s' 2026-03-09 19:10:03.719821 | controller | - ' ' 2026-03-09 19:10:03.719826 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.719830 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.719834 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.719838 | controller | - ' ' 2026-03-09 19:10:03.719843 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719847 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719851 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.719855 | controller | - ' ' 2026-03-09 19:10:03.719860 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.719864 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.719868 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.719872 | controller | available"' 2026-03-09 19:10:03.719877 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.719881 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.719885 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.719889 | controller | - ' exit 1' 2026-03-09 19:10:03.719894 | controller | - ' fi' 2026-03-09 19:10:03.719898 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.719902 | controller | - ' ' 2026-03-09 19:10:03.719906 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.719911 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.719930 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719935 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.719939 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719943 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.719947 | controller | -- \' 2026-03-09 19:10:03.719952 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.719958 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.719962 | controller | - ' logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.719966 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.719971 | controller | - ' logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.719976 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.719981 | controller | found' 2026-03-09 19:10:03.719986 | controller | - ' logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.719991 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.719995 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.719999 | controller | - ' ' 2026-03-09 19:10:03.720011 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.720015 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.720020 | controller | - ' ' 2026-03-09 19:10:03.720026 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.720031 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.720035 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.720039 | controller | - ' exit 1' 2026-03-09 19:10:03.720043 | controller | - ' fi' 2026-03-09 19:10:03.720048 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.720052 | controller | - ' ' 2026-03-09 19:10:03.720056 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.720060 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.720065 | controller | | base64 -d)' 2026-03-09 19:10:03.720069 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.720073 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.720077 | controller | - ' exit 1' 2026-03-09 19:10:03.720082 | controller | - ' fi' 2026-03-09 19:10:03.720086 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.720092 | controller | - ' ' 2026-03-09 19:10:03.720097 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.720101 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.720106 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.720110 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.720114 | controller | --timeout=300s' 2026-03-09 19:10:03.720118 | controller | - ' ' 2026-03-09 19:10:03.720123 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.720127 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.720131 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.720135 | controller | - ' ' 2026-03-09 19:10:03.720140 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720144 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720167 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720171 | controller | - ' ' 2026-03-09 19:10:03.720175 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.720180 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.720186 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.720191 | controller | available"' 2026-03-09 19:10:03.720195 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.720199 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.720203 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.720208 | controller | - ' exit 1' 2026-03-09 19:10:03.720212 | controller | - ' fi' 2026-03-09 19:10:03.720216 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.720221 | controller | - ' ' 2026-03-09 19:10:03.720225 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.720229 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.720233 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720238 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.720242 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720246 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.720250 | controller | -- \' 2026-03-09 19:10:03.720255 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720259 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.720266 | controller | - ' logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.720270 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.720275 | controller | - ' logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.720279 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.720283 | controller | found' 2026-03-09 19:10:03.720288 | controller | - ' logger.go:42: 18:53:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.720292 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.720296 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.720300 | controller | - ' ' 2026-03-09 19:10:03.720305 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.720317 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.720321 | controller | - ' ' 2026-03-09 19:10:03.720325 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.720330 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.720334 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.720338 | controller | - ' exit 1' 2026-03-09 19:10:03.720344 | controller | - ' fi' 2026-03-09 19:10:03.720349 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.720354 | controller | - ' ' 2026-03-09 19:10:03.720360 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.720365 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.720369 | controller | | base64 -d)' 2026-03-09 19:10:03.720373 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.720377 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.720382 | controller | - ' exit 1' 2026-03-09 19:10:03.720386 | controller | - ' fi' 2026-03-09 19:10:03.720390 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.720396 | controller | - ' ' 2026-03-09 19:10:03.720401 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.720405 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.720409 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.720413 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.720418 | controller | --timeout=300s' 2026-03-09 19:10:03.720424 | controller | - ' ' 2026-03-09 19:10:03.720428 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.720432 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.720436 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.720441 | controller | - ' ' 2026-03-09 19:10:03.720445 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720449 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720453 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720458 | controller | - ' ' 2026-03-09 19:10:03.720462 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.720466 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.720470 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.720475 | controller | available"' 2026-03-09 19:10:03.720481 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.720486 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.720490 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.720494 | controller | - ' exit 1' 2026-03-09 19:10:03.720499 | controller | - ' fi' 2026-03-09 19:10:03.720503 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.720507 | controller | - ' ' 2026-03-09 19:10:03.720511 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.720516 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.720520 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720524 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.720529 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720533 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.720537 | controller | -- \' 2026-03-09 19:10:03.720541 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720546 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.720550 | controller | - ' logger.go:42: 18:53:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.720555 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.720562 | controller | - ' logger.go:42: 18:53:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.720567 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.720571 | controller | found' 2026-03-09 19:10:03.720577 | controller | - ' logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.720582 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.720586 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.720590 | controller | - ' ' 2026-03-09 19:10:03.720595 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.720606 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.720610 | controller | - ' ' 2026-03-09 19:10:03.720615 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.720619 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.720623 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.720628 | controller | - ' exit 1' 2026-03-09 19:10:03.720632 | controller | - ' fi' 2026-03-09 19:10:03.720636 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.720644 | controller | - ' ' 2026-03-09 19:10:03.720648 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.720652 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.720657 | controller | | base64 -d)' 2026-03-09 19:10:03.720661 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.720665 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.720669 | controller | - ' exit 1' 2026-03-09 19:10:03.720674 | controller | - ' fi' 2026-03-09 19:10:03.720678 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.720682 | controller | - ' ' 2026-03-09 19:10:03.720687 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.720693 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.720697 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.720702 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.720706 | controller | --timeout=300s' 2026-03-09 19:10:03.720710 | controller | - ' ' 2026-03-09 19:10:03.720714 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.720719 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.720723 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.720727 | controller | - ' ' 2026-03-09 19:10:03.720735 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720740 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720744 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.720749 | controller | - ' ' 2026-03-09 19:10:03.720753 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.720757 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.720762 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.720766 | controller | available"' 2026-03-09 19:10:03.720770 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.720774 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.720779 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.720783 | controller | - ' exit 1' 2026-03-09 19:10:03.720789 | controller | - ' fi' 2026-03-09 19:10:03.720793 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.720798 | controller | - ' ' 2026-03-09 19:10:03.720802 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.720806 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.720810 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720815 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.720819 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720823 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.720828 | controller | -- \' 2026-03-09 19:10:03.720832 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.720836 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.720840 | controller | - ' logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.720845 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.720849 | controller | - ' logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.720853 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.720857 | controller | found' 2026-03-09 19:10:03.720862 | controller | - ' logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.720866 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.720870 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.720875 | controller | - ' ' 2026-03-09 19:10:03.720879 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.720883 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.720887 | controller | - ' ' 2026-03-09 19:10:03.720898 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.720903 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.720907 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.720912 | controller | - ' exit 1' 2026-03-09 19:10:03.720916 | controller | - ' fi' 2026-03-09 19:10:03.720920 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.720925 | controller | - ' ' 2026-03-09 19:10:03.720929 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.720933 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.720937 | controller | | base64 -d)' 2026-03-09 19:10:03.720942 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.720946 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.720950 | controller | - ' exit 1' 2026-03-09 19:10:03.720955 | controller | - ' fi' 2026-03-09 19:10:03.720959 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.720963 | controller | - ' ' 2026-03-09 19:10:03.720967 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.720972 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.720976 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.720980 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.720984 | controller | --timeout=300s' 2026-03-09 19:10:03.720993 | controller | - ' ' 2026-03-09 19:10:03.720997 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.721001 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.721006 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.721010 | controller | - ' ' 2026-03-09 19:10:03.721014 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721018 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721023 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721027 | controller | - ' ' 2026-03-09 19:10:03.721031 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.721035 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.721040 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.721044 | controller | available"' 2026-03-09 19:10:03.721048 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.721053 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.721057 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.721061 | controller | - ' exit 1' 2026-03-09 19:10:03.721065 | controller | - ' fi' 2026-03-09 19:10:03.721070 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.721074 | controller | - ' ' 2026-03-09 19:10:03.721078 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.721082 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.721087 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721091 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.721095 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721100 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.721104 | controller | -- \' 2026-03-09 19:10:03.721108 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721112 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.721117 | controller | - ' logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.721121 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.721125 | controller | - ' logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.721156 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.721164 | controller | found' 2026-03-09 19:10:03.721169 | controller | - ' logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.721174 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.721179 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.721184 | controller | - ' ' 2026-03-09 19:10:03.721188 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.721192 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.721196 | controller | - ' ' 2026-03-09 19:10:03.721207 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.721214 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.721218 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.721223 | controller | - ' exit 1' 2026-03-09 19:10:03.721227 | controller | - ' fi' 2026-03-09 19:10:03.721231 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.721236 | controller | - ' ' 2026-03-09 19:10:03.721240 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.721244 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.721249 | controller | | base64 -d)' 2026-03-09 19:10:03.721253 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.721257 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.721261 | controller | - ' exit 1' 2026-03-09 19:10:03.721266 | controller | - ' fi' 2026-03-09 19:10:03.721270 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.721274 | controller | - ' ' 2026-03-09 19:10:03.721279 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.721283 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.721289 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.721293 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.721298 | controller | --timeout=300s' 2026-03-09 19:10:03.721302 | controller | - ' ' 2026-03-09 19:10:03.721306 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.721310 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.721315 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.721319 | controller | - ' ' 2026-03-09 19:10:03.721323 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721327 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721332 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721336 | controller | - ' ' 2026-03-09 19:10:03.721340 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.721344 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.721349 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.721353 | controller | available"' 2026-03-09 19:10:03.721357 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.721361 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.721366 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.721370 | controller | - ' exit 1' 2026-03-09 19:10:03.721374 | controller | - ' fi' 2026-03-09 19:10:03.721378 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.721383 | controller | - ' ' 2026-03-09 19:10:03.721387 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.721391 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.721395 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721400 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.721406 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721410 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.721414 | controller | -- \' 2026-03-09 19:10:03.721419 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721423 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.721427 | controller | - ' logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.721431 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.721438 | controller | - ' logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.721442 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.721446 | controller | found' 2026-03-09 19:10:03.721451 | controller | - ' logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.721455 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.721459 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.721464 | controller | - ' ' 2026-03-09 19:10:03.721468 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.721472 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.721476 | controller | - ' ' 2026-03-09 19:10:03.721481 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.721492 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.721497 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.721501 | controller | - ' exit 1' 2026-03-09 19:10:03.721506 | controller | - ' fi' 2026-03-09 19:10:03.721510 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.721514 | controller | - ' ' 2026-03-09 19:10:03.721518 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.721523 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.721527 | controller | | base64 -d)' 2026-03-09 19:10:03.721531 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.721535 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.721540 | controller | - ' exit 1' 2026-03-09 19:10:03.721544 | controller | - ' fi' 2026-03-09 19:10:03.721548 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.721552 | controller | - ' ' 2026-03-09 19:10:03.721557 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.721561 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.721565 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.721569 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.721573 | controller | --timeout=300s' 2026-03-09 19:10:03.721578 | controller | - ' ' 2026-03-09 19:10:03.721582 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.721586 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.721591 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.721595 | controller | - ' ' 2026-03-09 19:10:03.721601 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721608 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721612 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721616 | controller | - ' ' 2026-03-09 19:10:03.721621 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.721625 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.721629 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.721633 | controller | available"' 2026-03-09 19:10:03.721638 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.721643 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.721648 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.721652 | controller | - ' exit 1' 2026-03-09 19:10:03.721656 | controller | - ' fi' 2026-03-09 19:10:03.721661 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.721665 | controller | - ' ' 2026-03-09 19:10:03.721669 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.721673 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.721678 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721682 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.721686 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721690 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.721695 | controller | -- \' 2026-03-09 19:10:03.721699 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721703 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.721707 | controller | - ' logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.721712 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.721716 | controller | - ' logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.721720 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.721724 | controller | found' 2026-03-09 19:10:03.721729 | controller | - ' logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.721733 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.721737 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.721741 | controller | - ' ' 2026-03-09 19:10:03.721746 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.721750 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.721754 | controller | - ' ' 2026-03-09 19:10:03.721758 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.721763 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.721774 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.721778 | controller | - ' exit 1' 2026-03-09 19:10:03.721783 | controller | - ' fi' 2026-03-09 19:10:03.721787 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.721791 | controller | - ' ' 2026-03-09 19:10:03.721796 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.721802 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.721806 | controller | | base64 -d)' 2026-03-09 19:10:03.721810 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.721815 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.721819 | controller | - ' exit 1' 2026-03-09 19:10:03.721825 | controller | - ' fi' 2026-03-09 19:10:03.721829 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.721833 | controller | - ' ' 2026-03-09 19:10:03.721838 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.721842 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.721846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.721853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.721858 | controller | --timeout=300s' 2026-03-09 19:10:03.721862 | controller | - ' ' 2026-03-09 19:10:03.721866 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.721870 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.721875 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.721879 | controller | - ' ' 2026-03-09 19:10:03.721883 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721887 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721892 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.721896 | controller | - ' ' 2026-03-09 19:10:03.721900 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.721904 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.721909 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.721913 | controller | available"' 2026-03-09 19:10:03.721917 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.721921 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.721926 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.721930 | controller | - ' exit 1' 2026-03-09 19:10:03.721934 | controller | - ' fi' 2026-03-09 19:10:03.721938 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.721943 | controller | - ' ' 2026-03-09 19:10:03.721947 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.721951 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.721955 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721960 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.721964 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721968 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.721972 | controller | -- \' 2026-03-09 19:10:03.721977 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.721981 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.721987 | controller | - ' logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.721991 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.721995 | controller | - ' logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.722000 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.722004 | controller | found' 2026-03-09 19:10:03.722008 | controller | - ' logger.go:42: 18:54:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.722012 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.722017 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.722021 | controller | - ' ' 2026-03-09 19:10:03.722025 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.722029 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.722034 | controller | - ' ' 2026-03-09 19:10:03.722038 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.722042 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.722047 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.722051 | controller | - ' exit 1' 2026-03-09 19:10:03.722062 | controller | - ' fi' 2026-03-09 19:10:03.722067 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.722071 | controller | - ' ' 2026-03-09 19:10:03.722075 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.722080 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.722084 | controller | | base64 -d)' 2026-03-09 19:10:03.722088 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.722092 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.722097 | controller | - ' exit 1' 2026-03-09 19:10:03.722101 | controller | - ' fi' 2026-03-09 19:10:03.722105 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.722110 | controller | - ' ' 2026-03-09 19:10:03.722114 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.722118 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.722122 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.722127 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.722131 | controller | --timeout=300s' 2026-03-09 19:10:03.722135 | controller | - ' ' 2026-03-09 19:10:03.722140 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.722144 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.722160 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.722165 | controller | - ' ' 2026-03-09 19:10:03.722169 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722173 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722178 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722182 | controller | - ' ' 2026-03-09 19:10:03.722186 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.722190 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.722196 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.722201 | controller | available"' 2026-03-09 19:10:03.722205 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.722209 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.722214 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.722218 | controller | - ' exit 1' 2026-03-09 19:10:03.722222 | controller | - ' fi' 2026-03-09 19:10:03.722226 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.722231 | controller | - ' ' 2026-03-09 19:10:03.722235 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.722239 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.722243 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722248 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.722252 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722256 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.722260 | controller | -- \' 2026-03-09 19:10:03.722265 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722269 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.722273 | controller | - ' logger.go:42: 18:54:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.722277 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.722282 | controller | - ' logger.go:42: 18:54:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.722286 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.722290 | controller | found' 2026-03-09 19:10:03.722294 | controller | - ' logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.722299 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.722303 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.722307 | controller | - ' ' 2026-03-09 19:10:03.722311 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.722316 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.722320 | controller | - ' ' 2026-03-09 19:10:03.722324 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.722328 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.722333 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.722337 | controller | - ' exit 1' 2026-03-09 19:10:03.722341 | controller | - ' fi' 2026-03-09 19:10:03.722353 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.722358 | controller | - ' ' 2026-03-09 19:10:03.722362 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.722366 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.722371 | controller | | base64 -d)' 2026-03-09 19:10:03.722375 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.722379 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.722384 | controller | - ' exit 1' 2026-03-09 19:10:03.722388 | controller | - ' fi' 2026-03-09 19:10:03.722392 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.722396 | controller | - ' ' 2026-03-09 19:10:03.722402 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.722407 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.722411 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.722415 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.722419 | controller | --timeout=300s' 2026-03-09 19:10:03.722424 | controller | - ' ' 2026-03-09 19:10:03.722428 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.722432 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.722437 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.722441 | controller | - ' ' 2026-03-09 19:10:03.722445 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722449 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722454 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722458 | controller | - ' ' 2026-03-09 19:10:03.722464 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.722468 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.722473 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.722477 | controller | available"' 2026-03-09 19:10:03.722481 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.722486 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.722490 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.722494 | controller | - ' exit 1' 2026-03-09 19:10:03.722498 | controller | - ' fi' 2026-03-09 19:10:03.722503 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.722507 | controller | - ' ' 2026-03-09 19:10:03.722511 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.722515 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.722520 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722524 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.722528 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722532 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.722538 | controller | -- \' 2026-03-09 19:10:03.722543 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722547 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.722551 | controller | - ' logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.722555 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.722560 | controller | - ' logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.722564 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.722568 | controller | found' 2026-03-09 19:10:03.722572 | controller | - ' logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.722578 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.722583 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.722587 | controller | - ' ' 2026-03-09 19:10:03.722591 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.722596 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.722600 | controller | - ' ' 2026-03-09 19:10:03.722604 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.722608 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.722613 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.722617 | controller | - ' exit 1' 2026-03-09 19:10:03.722621 | controller | - ' fi' 2026-03-09 19:10:03.722626 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.722630 | controller | - ' ' 2026-03-09 19:10:03.722641 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.722646 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.722650 | controller | | base64 -d)' 2026-03-09 19:10:03.722655 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.722659 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.722663 | controller | - ' exit 1' 2026-03-09 19:10:03.722667 | controller | - ' fi' 2026-03-09 19:10:03.722672 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.722676 | controller | - ' ' 2026-03-09 19:10:03.722680 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.722684 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.722689 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.722693 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.722697 | controller | --timeout=300s' 2026-03-09 19:10:03.722701 | controller | - ' ' 2026-03-09 19:10:03.722706 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.722710 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.722714 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.722718 | controller | - ' ' 2026-03-09 19:10:03.722723 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722727 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722731 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.722735 | controller | - ' ' 2026-03-09 19:10:03.722740 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.722744 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.722748 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.722753 | controller | available"' 2026-03-09 19:10:03.722757 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.722761 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.722765 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.722770 | controller | - ' exit 1' 2026-03-09 19:10:03.722774 | controller | - ' fi' 2026-03-09 19:10:03.722778 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.722784 | controller | - ' ' 2026-03-09 19:10:03.722788 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.722793 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.722797 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722801 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.722805 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722810 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.722814 | controller | -- \' 2026-03-09 19:10:03.722818 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.722823 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.722827 | controller | - ' logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.722831 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.722835 | controller | - ' logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.722840 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.722844 | controller | found' 2026-03-09 19:10:03.722848 | controller | - ' logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.722853 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.722857 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.722861 | controller | - ' ' 2026-03-09 19:10:03.722865 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.722870 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.722874 | controller | - ' ' 2026-03-09 19:10:03.722880 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.722884 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.722889 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.722893 | controller | - ' exit 1' 2026-03-09 19:10:03.722897 | controller | - ' fi' 2026-03-09 19:10:03.722901 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.722906 | controller | - ' ' 2026-03-09 19:10:03.722912 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.722924 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.722929 | controller | | base64 -d)' 2026-03-09 19:10:03.722933 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.722938 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.722942 | controller | - ' exit 1' 2026-03-09 19:10:03.722946 | controller | - ' fi' 2026-03-09 19:10:03.722951 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.722955 | controller | - ' ' 2026-03-09 19:10:03.722959 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.722963 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.722967 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.722972 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.722976 | controller | --timeout=300s' 2026-03-09 19:10:03.722980 | controller | - ' ' 2026-03-09 19:10:03.722985 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.722990 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.722995 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.722999 | controller | - ' ' 2026-03-09 19:10:03.723003 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723008 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723012 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723016 | controller | - ' ' 2026-03-09 19:10:03.723021 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.723025 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.723031 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.723035 | controller | available"' 2026-03-09 19:10:03.723039 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.723043 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.723048 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.723052 | controller | - ' exit 1' 2026-03-09 19:10:03.723056 | controller | - ' fi' 2026-03-09 19:10:03.723060 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.723065 | controller | - ' ' 2026-03-09 19:10:03.723069 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.723073 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.723078 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723082 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.723086 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723090 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.723095 | controller | -- \' 2026-03-09 19:10:03.723099 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723103 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.723107 | controller | - ' logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.723112 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.723116 | controller | - ' logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.723120 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.723124 | controller | found' 2026-03-09 19:10:03.723129 | controller | - ' logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.723133 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.723137 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.723141 | controller | - ' ' 2026-03-09 19:10:03.723156 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.723162 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.723167 | controller | - ' ' 2026-03-09 19:10:03.723171 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.723177 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.723181 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.723186 | controller | - ' exit 1' 2026-03-09 19:10:03.723190 | controller | - ' fi' 2026-03-09 19:10:03.723194 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.723198 | controller | - ' ' 2026-03-09 19:10:03.723203 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.723214 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.723219 | controller | | base64 -d)' 2026-03-09 19:10:03.723223 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.723227 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.723232 | controller | - ' exit 1' 2026-03-09 19:10:03.723236 | controller | - ' fi' 2026-03-09 19:10:03.723240 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.723244 | controller | - ' ' 2026-03-09 19:10:03.723249 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.723253 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.723257 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.723261 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.723266 | controller | --timeout=300s' 2026-03-09 19:10:03.723270 | controller | - ' ' 2026-03-09 19:10:03.723274 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.723278 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.723283 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.723287 | controller | - ' ' 2026-03-09 19:10:03.723291 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723295 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723300 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723304 | controller | - ' ' 2026-03-09 19:10:03.723308 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.723312 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.723317 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.723321 | controller | available"' 2026-03-09 19:10:03.723325 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.723329 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.723334 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.723340 | controller | - ' exit 1' 2026-03-09 19:10:03.723344 | controller | - ' fi' 2026-03-09 19:10:03.723348 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.723353 | controller | - ' ' 2026-03-09 19:10:03.723357 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.723361 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.723365 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723370 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.723377 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723381 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.723386 | controller | -- \' 2026-03-09 19:10:03.723390 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723394 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.723398 | controller | - ' logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.723403 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.723407 | controller | - ' logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.723411 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.723415 | controller | found' 2026-03-09 19:10:03.723420 | controller | - ' logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.723424 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.723428 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.723432 | controller | - ' ' 2026-03-09 19:10:03.723437 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.723441 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.723445 | controller | - ' ' 2026-03-09 19:10:03.723449 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.723454 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.723458 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.723462 | controller | - ' exit 1' 2026-03-09 19:10:03.723466 | controller | - ' fi' 2026-03-09 19:10:03.723471 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.723475 | controller | - ' ' 2026-03-09 19:10:03.723479 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.723484 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.723495 | controller | | base64 -d)' 2026-03-09 19:10:03.723500 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.723504 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.723508 | controller | - ' exit 1' 2026-03-09 19:10:03.723513 | controller | - ' fi' 2026-03-09 19:10:03.723517 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.723521 | controller | - ' ' 2026-03-09 19:10:03.723525 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.723530 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.723534 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.723538 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.723542 | controller | --timeout=300s' 2026-03-09 19:10:03.723547 | controller | - ' ' 2026-03-09 19:10:03.723551 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.723555 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.723559 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.723564 | controller | - ' ' 2026-03-09 19:10:03.723568 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723572 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723580 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723584 | controller | - ' ' 2026-03-09 19:10:03.723588 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.723593 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.723597 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.723601 | controller | available"' 2026-03-09 19:10:03.723606 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.723611 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.723616 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.723620 | controller | - ' exit 1' 2026-03-09 19:10:03.723624 | controller | - ' fi' 2026-03-09 19:10:03.723629 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.723633 | controller | - ' ' 2026-03-09 19:10:03.723637 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.723641 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.723646 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723650 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.723654 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723658 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.723663 | controller | -- \' 2026-03-09 19:10:03.723667 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723671 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.723676 | controller | - ' logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.723680 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.723684 | controller | - ' logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.723688 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.723693 | controller | found' 2026-03-09 19:10:03.723699 | controller | - ' logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.723712 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.723718 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.723724 | controller | - ' ' 2026-03-09 19:10:03.723728 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.723733 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.723737 | controller | - ' ' 2026-03-09 19:10:03.723741 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.723745 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.723750 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.723754 | controller | - ' exit 1' 2026-03-09 19:10:03.723758 | controller | - ' fi' 2026-03-09 19:10:03.723762 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.723767 | controller | - ' ' 2026-03-09 19:10:03.723771 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.723775 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.723782 | controller | | base64 -d)' 2026-03-09 19:10:03.723793 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.723797 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.723802 | controller | - ' exit 1' 2026-03-09 19:10:03.723806 | controller | - ' fi' 2026-03-09 19:10:03.723810 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.723815 | controller | - ' ' 2026-03-09 19:10:03.723819 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.723823 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.723827 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.723832 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.723836 | controller | --timeout=300s' 2026-03-09 19:10:03.723840 | controller | - ' ' 2026-03-09 19:10:03.723844 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.723849 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.723853 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.723857 | controller | - ' ' 2026-03-09 19:10:03.723861 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723866 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723870 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.723874 | controller | - ' ' 2026-03-09 19:10:03.723878 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.723883 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.723887 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.723891 | controller | available"' 2026-03-09 19:10:03.723898 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.723902 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.723906 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.723911 | controller | - ' exit 1' 2026-03-09 19:10:03.723915 | controller | - ' fi' 2026-03-09 19:10:03.723919 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.723924 | controller | - ' ' 2026-03-09 19:10:03.723928 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.723932 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.723936 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723941 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.723945 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723949 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.723953 | controller | -- \' 2026-03-09 19:10:03.723958 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.723962 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.723968 | controller | - ' logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.723972 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.723977 | controller | - ' logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.723981 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.723985 | controller | found' 2026-03-09 19:10:03.723990 | controller | - ' logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.723994 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.723998 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.724002 | controller | - ' ' 2026-03-09 19:10:03.724007 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.724011 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.724015 | controller | - ' ' 2026-03-09 19:10:03.724019 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.724024 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.724028 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.724032 | controller | - ' exit 1' 2026-03-09 19:10:03.724037 | controller | - ' fi' 2026-03-09 19:10:03.724041 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.724045 | controller | - ' ' 2026-03-09 19:10:03.724049 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.724054 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.724058 | controller | | base64 -d)' 2026-03-09 19:10:03.724062 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.724073 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.724078 | controller | - ' exit 1' 2026-03-09 19:10:03.724082 | controller | - ' fi' 2026-03-09 19:10:03.724086 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.724091 | controller | - ' ' 2026-03-09 19:10:03.724095 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.724099 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.724104 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.724108 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.724112 | controller | --timeout=300s' 2026-03-09 19:10:03.724116 | controller | - ' ' 2026-03-09 19:10:03.724121 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.724125 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.724129 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.724133 | controller | - ' ' 2026-03-09 19:10:03.724138 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724142 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724162 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724169 | controller | - ' ' 2026-03-09 19:10:03.724173 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.724177 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.724182 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.724188 | controller | available"' 2026-03-09 19:10:03.724193 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.724200 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.724204 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.724208 | controller | - ' exit 1' 2026-03-09 19:10:03.724213 | controller | - ' fi' 2026-03-09 19:10:03.724217 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.724221 | controller | - ' ' 2026-03-09 19:10:03.724227 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.724232 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.724236 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724240 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.724244 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724249 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.724253 | controller | -- \' 2026-03-09 19:10:03.724257 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724262 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.724267 | controller | - ' logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.724272 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.724276 | controller | - ' logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.724280 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.724285 | controller | found' 2026-03-09 19:10:03.724289 | controller | - ' logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.724293 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.724298 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.724302 | controller | - ' ' 2026-03-09 19:10:03.724306 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.724310 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.724315 | controller | - ' ' 2026-03-09 19:10:03.724319 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.724323 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.724328 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.724332 | controller | - ' exit 1' 2026-03-09 19:10:03.724336 | controller | - ' fi' 2026-03-09 19:10:03.724341 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.724345 | controller | - ' ' 2026-03-09 19:10:03.724349 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.724353 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.724358 | controller | | base64 -d)' 2026-03-09 19:10:03.724362 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.724374 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.724379 | controller | - ' exit 1' 2026-03-09 19:10:03.724384 | controller | - ' fi' 2026-03-09 19:10:03.724388 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.724392 | controller | - ' ' 2026-03-09 19:10:03.724396 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.724403 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.724407 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.724411 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.724415 | controller | --timeout=300s' 2026-03-09 19:10:03.724422 | controller | - ' ' 2026-03-09 19:10:03.724426 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.724430 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.724434 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.724439 | controller | - ' ' 2026-03-09 19:10:03.724443 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724447 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724451 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724456 | controller | - ' ' 2026-03-09 19:10:03.724460 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.724464 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.724469 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.724473 | controller | available"' 2026-03-09 19:10:03.724477 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.724481 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.724486 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.724490 | controller | - ' exit 1' 2026-03-09 19:10:03.724494 | controller | - ' fi' 2026-03-09 19:10:03.724499 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.724503 | controller | - ' ' 2026-03-09 19:10:03.724507 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.724511 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.724516 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724520 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.724524 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724528 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.724533 | controller | -- \' 2026-03-09 19:10:03.724537 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724541 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.724546 | controller | - ' logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.724550 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.724554 | controller | - ' logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.724558 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.724563 | controller | found' 2026-03-09 19:10:03.724567 | controller | - ' logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.724571 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.724577 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.724582 | controller | - ' ' 2026-03-09 19:10:03.724586 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.724590 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.724595 | controller | - ' ' 2026-03-09 19:10:03.724599 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.724603 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.724608 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.724612 | controller | - ' exit 1' 2026-03-09 19:10:03.724616 | controller | - ' fi' 2026-03-09 19:10:03.724621 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.724625 | controller | - ' ' 2026-03-09 19:10:03.724629 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.724633 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.724638 | controller | | base64 -d)' 2026-03-09 19:10:03.724642 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.724646 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.724656 | controller | - ' exit 1' 2026-03-09 19:10:03.724668 | controller | - ' fi' 2026-03-09 19:10:03.724673 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.724677 | controller | - ' ' 2026-03-09 19:10:03.724682 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.724686 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.724690 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.724695 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.724699 | controller | --timeout=300s' 2026-03-09 19:10:03.724703 | controller | - ' ' 2026-03-09 19:10:03.724707 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.724712 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.724716 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.724720 | controller | - ' ' 2026-03-09 19:10:03.724725 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724731 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724736 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.724740 | controller | - ' ' 2026-03-09 19:10:03.724745 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.724749 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.724753 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.724757 | controller | available"' 2026-03-09 19:10:03.724762 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.724766 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.724770 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.724774 | controller | - ' exit 1' 2026-03-09 19:10:03.724779 | controller | - ' fi' 2026-03-09 19:10:03.724783 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.724791 | controller | - ' ' 2026-03-09 19:10:03.724795 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.724800 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.724804 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724810 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.724814 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724819 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.724823 | controller | -- \' 2026-03-09 19:10:03.724827 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.724831 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.724836 | controller | - ' logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.724840 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.724844 | controller | - ' logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.724848 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.724853 | controller | found' 2026-03-09 19:10:03.724857 | controller | - ' logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.724861 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.724865 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.724872 | controller | - ' ' 2026-03-09 19:10:03.724876 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.724880 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.724885 | controller | - ' ' 2026-03-09 19:10:03.724889 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.724893 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.724898 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.724902 | controller | - ' exit 1' 2026-03-09 19:10:03.724907 | controller | - ' fi' 2026-03-09 19:10:03.724911 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.724915 | controller | - ' ' 2026-03-09 19:10:03.724920 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.724924 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.724928 | controller | | base64 -d)' 2026-03-09 19:10:03.724932 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.724937 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.724941 | controller | - ' exit 1' 2026-03-09 19:10:03.724945 | controller | - ' fi' 2026-03-09 19:10:03.724957 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.724961 | controller | - ' ' 2026-03-09 19:10:03.724967 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.724972 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.724976 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.724981 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.724985 | controller | --timeout=300s' 2026-03-09 19:10:03.724989 | controller | - ' ' 2026-03-09 19:10:03.724994 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.725000 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.725004 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.725008 | controller | - ' ' 2026-03-09 19:10:03.725013 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725017 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725021 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725026 | controller | - ' ' 2026-03-09 19:10:03.725030 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.725034 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.725039 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.725043 | controller | available"' 2026-03-09 19:10:03.725047 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.725052 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.725056 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.725060 | controller | - ' exit 1' 2026-03-09 19:10:03.725065 | controller | - ' fi' 2026-03-09 19:10:03.725069 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.725073 | controller | - ' ' 2026-03-09 19:10:03.725078 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.725082 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.725086 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725090 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.725095 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725099 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.725103 | controller | -- \' 2026-03-09 19:10:03.725108 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725112 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.725116 | controller | - ' logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.725121 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.725125 | controller | - ' logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.725129 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.725134 | controller | found' 2026-03-09 19:10:03.725138 | controller | - ' logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.725142 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.725160 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.725165 | controller | - ' ' 2026-03-09 19:10:03.725170 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.725174 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.725178 | controller | - ' ' 2026-03-09 19:10:03.725182 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.725187 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.725193 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.725197 | controller | - ' exit 1' 2026-03-09 19:10:03.725202 | controller | - ' fi' 2026-03-09 19:10:03.725206 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.725210 | controller | - ' ' 2026-03-09 19:10:03.725215 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.725219 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.725223 | controller | | base64 -d)' 2026-03-09 19:10:03.725227 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.725232 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.725236 | controller | - ' exit 1' 2026-03-09 19:10:03.725240 | controller | - ' fi' 2026-03-09 19:10:03.725245 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.725249 | controller | - ' ' 2026-03-09 19:10:03.725260 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.725265 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.725269 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.725274 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.725278 | controller | --timeout=300s' 2026-03-09 19:10:03.725282 | controller | - ' ' 2026-03-09 19:10:03.725287 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.725291 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.725295 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.725299 | controller | - ' ' 2026-03-09 19:10:03.725304 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725308 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725312 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725316 | controller | - ' ' 2026-03-09 19:10:03.725321 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.725325 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.725329 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.725333 | controller | available"' 2026-03-09 19:10:03.725338 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.725342 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.725346 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.725350 | controller | - ' exit 1' 2026-03-09 19:10:03.725355 | controller | - ' fi' 2026-03-09 19:10:03.725359 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.725363 | controller | - ' ' 2026-03-09 19:10:03.725367 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.725372 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.725376 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725380 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.725384 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725391 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.725395 | controller | -- \' 2026-03-09 19:10:03.725399 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725405 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.725410 | controller | - ' logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.725414 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.725418 | controller | - ' logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.725422 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.725427 | controller | found' 2026-03-09 19:10:03.725431 | controller | - ' logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.725435 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.725439 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.725444 | controller | - ' ' 2026-03-09 19:10:03.725448 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.725452 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.725456 | controller | - ' ' 2026-03-09 19:10:03.725461 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.725465 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.725469 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.725473 | controller | - ' exit 1' 2026-03-09 19:10:03.725478 | controller | - ' fi' 2026-03-09 19:10:03.725482 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.725486 | controller | - ' ' 2026-03-09 19:10:03.725493 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.725497 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.725501 | controller | | base64 -d)' 2026-03-09 19:10:03.725506 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.725510 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.725514 | controller | - ' exit 1' 2026-03-09 19:10:03.725519 | controller | - ' fi' 2026-03-09 19:10:03.725523 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.725527 | controller | - ' ' 2026-03-09 19:10:03.725531 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.725542 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.725547 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.725551 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.725555 | controller | --timeout=300s' 2026-03-09 19:10:03.725560 | controller | - ' ' 2026-03-09 19:10:03.725564 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.725568 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.725573 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.725577 | controller | - ' ' 2026-03-09 19:10:03.725583 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725587 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725593 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725597 | controller | - ' ' 2026-03-09 19:10:03.725601 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.725607 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.725611 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.725616 | controller | available"' 2026-03-09 19:10:03.725620 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.725624 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.725629 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.725633 | controller | - ' exit 1' 2026-03-09 19:10:03.725637 | controller | - ' fi' 2026-03-09 19:10:03.725641 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.725646 | controller | - ' ' 2026-03-09 19:10:03.725650 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.725654 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.725658 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725663 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.725667 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725671 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.725675 | controller | -- \' 2026-03-09 19:10:03.725680 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725684 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.725688 | controller | - ' logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.725693 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.725697 | controller | - ' logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.725703 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.725707 | controller | found' 2026-03-09 19:10:03.725711 | controller | - ' logger.go:42: 18:54:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.725716 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.725720 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.725724 | controller | - ' ' 2026-03-09 19:10:03.725729 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.725733 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.725737 | controller | - ' ' 2026-03-09 19:10:03.725741 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.725746 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.725750 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.725754 | controller | - ' exit 1' 2026-03-09 19:10:03.725758 | controller | - ' fi' 2026-03-09 19:10:03.725763 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.725767 | controller | - ' ' 2026-03-09 19:10:03.725771 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.725775 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.725780 | controller | | base64 -d)' 2026-03-09 19:10:03.725786 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.725790 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.725794 | controller | - ' exit 1' 2026-03-09 19:10:03.725798 | controller | - ' fi' 2026-03-09 19:10:03.725803 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.725807 | controller | - ' ' 2026-03-09 19:10:03.725811 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.725823 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.725827 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.725832 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.725836 | controller | --timeout=300s' 2026-03-09 19:10:03.725840 | controller | - ' ' 2026-03-09 19:10:03.725845 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.725849 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.725853 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.725857 | controller | - ' ' 2026-03-09 19:10:03.725862 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725866 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725870 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.725875 | controller | - ' ' 2026-03-09 19:10:03.725879 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.725883 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.725887 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.725892 | controller | available"' 2026-03-09 19:10:03.725896 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.725900 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.725904 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.725909 | controller | - ' exit 1' 2026-03-09 19:10:03.725913 | controller | - ' fi' 2026-03-09 19:10:03.725917 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.725921 | controller | - ' ' 2026-03-09 19:10:03.725926 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.725930 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.725934 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725938 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.725943 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725947 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.725951 | controller | -- \' 2026-03-09 19:10:03.725955 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.725960 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.725964 | controller | - ' logger.go:42: 18:54:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.725970 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.725978 | controller | - ' logger.go:42: 18:54:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.725984 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.725989 | controller | found' 2026-03-09 19:10:03.725993 | controller | - ' logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.725998 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.726002 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.726006 | controller | - ' ' 2026-03-09 19:10:03.726011 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.726015 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.726019 | controller | - ' ' 2026-03-09 19:10:03.726023 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.726028 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.726032 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.726036 | controller | - ' exit 1' 2026-03-09 19:10:03.726040 | controller | - ' fi' 2026-03-09 19:10:03.726047 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.726051 | controller | - ' ' 2026-03-09 19:10:03.726055 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.726059 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.726064 | controller | | base64 -d)' 2026-03-09 19:10:03.726068 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.726072 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.726076 | controller | - ' exit 1' 2026-03-09 19:10:03.726081 | controller | - ' fi' 2026-03-09 19:10:03.726086 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.726091 | controller | - ' ' 2026-03-09 19:10:03.726096 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.726100 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.726112 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.726116 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.726120 | controller | --timeout=300s' 2026-03-09 19:10:03.726125 | controller | - ' ' 2026-03-09 19:10:03.726129 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.726133 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.726138 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.726142 | controller | - ' ' 2026-03-09 19:10:03.726156 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726162 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726167 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726171 | controller | - ' ' 2026-03-09 19:10:03.726175 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.726179 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.726184 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.726188 | controller | available"' 2026-03-09 19:10:03.726195 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.726200 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.726204 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.726208 | controller | - ' exit 1' 2026-03-09 19:10:03.726213 | controller | - ' fi' 2026-03-09 19:10:03.726217 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.726221 | controller | - ' ' 2026-03-09 19:10:03.726225 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.726230 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.726236 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726241 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.726247 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726251 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.726256 | controller | -- \' 2026-03-09 19:10:03.726260 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726264 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.726268 | controller | - ' logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.726273 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.726277 | controller | - ' logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.726281 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.726286 | controller | found' 2026-03-09 19:10:03.726290 | controller | - ' logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.726294 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.726299 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.726303 | controller | - ' ' 2026-03-09 19:10:03.726307 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.726311 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.726316 | controller | - ' ' 2026-03-09 19:10:03.726320 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.726324 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.726328 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.726333 | controller | - ' exit 1' 2026-03-09 19:10:03.726339 | controller | - ' fi' 2026-03-09 19:10:03.726344 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.726348 | controller | - ' ' 2026-03-09 19:10:03.726353 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.726357 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.726361 | controller | | base64 -d)' 2026-03-09 19:10:03.726365 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.726370 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.726374 | controller | - ' exit 1' 2026-03-09 19:10:03.726378 | controller | - ' fi' 2026-03-09 19:10:03.726383 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.726387 | controller | - ' ' 2026-03-09 19:10:03.726391 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.726396 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.726411 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.726416 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.726421 | controller | --timeout=300s' 2026-03-09 19:10:03.726425 | controller | - ' ' 2026-03-09 19:10:03.726429 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.726433 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.726438 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.726442 | controller | - ' ' 2026-03-09 19:10:03.726446 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726451 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726455 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726459 | controller | - ' ' 2026-03-09 19:10:03.726463 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.726468 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.726472 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.726477 | controller | available"' 2026-03-09 19:10:03.726482 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.726487 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.726492 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.726496 | controller | - ' exit 1' 2026-03-09 19:10:03.726500 | controller | - ' fi' 2026-03-09 19:10:03.726504 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.726509 | controller | - ' ' 2026-03-09 19:10:03.726513 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.726517 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.726521 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726526 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.726530 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726534 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.726539 | controller | -- \' 2026-03-09 19:10:03.726543 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726547 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.726551 | controller | - ' logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.726556 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.726560 | controller | - ' logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.726567 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.726571 | controller | found' 2026-03-09 19:10:03.726575 | controller | - ' logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.726580 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.726584 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.726592 | controller | - ' ' 2026-03-09 19:10:03.726597 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.726601 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.726605 | controller | - ' ' 2026-03-09 19:10:03.726609 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.726614 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.726618 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.726622 | controller | - ' exit 1' 2026-03-09 19:10:03.726626 | controller | - ' fi' 2026-03-09 19:10:03.726631 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.726635 | controller | - ' ' 2026-03-09 19:10:03.726639 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.726643 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.726648 | controller | | base64 -d)' 2026-03-09 19:10:03.726652 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.726656 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.726661 | controller | - ' exit 1' 2026-03-09 19:10:03.726665 | controller | - ' fi' 2026-03-09 19:10:03.726669 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.726673 | controller | - ' ' 2026-03-09 19:10:03.726678 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.726682 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.726686 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.726697 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.726702 | controller | --timeout=300s' 2026-03-09 19:10:03.726706 | controller | - ' ' 2026-03-09 19:10:03.726710 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.726715 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.726721 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.726725 | controller | - ' ' 2026-03-09 19:10:03.726730 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726734 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726738 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.726742 | controller | - ' ' 2026-03-09 19:10:03.726747 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.726751 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.726755 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.726760 | controller | available"' 2026-03-09 19:10:03.726764 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.726770 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.726774 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.726778 | controller | - ' exit 1' 2026-03-09 19:10:03.726783 | controller | - ' fi' 2026-03-09 19:10:03.726787 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.726791 | controller | - ' ' 2026-03-09 19:10:03.726795 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.726801 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.726806 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726810 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.726814 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726819 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.726823 | controller | -- \' 2026-03-09 19:10:03.726827 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.726831 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.726836 | controller | - ' logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.726840 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.726844 | controller | - ' logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.726848 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.726853 | controller | found' 2026-03-09 19:10:03.726857 | controller | - ' logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.726861 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.726866 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.726870 | controller | - ' ' 2026-03-09 19:10:03.726874 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.726879 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.726883 | controller | - ' ' 2026-03-09 19:10:03.726887 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.726891 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.726896 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.726900 | controller | - ' exit 1' 2026-03-09 19:10:03.726904 | controller | - ' fi' 2026-03-09 19:10:03.726908 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.726913 | controller | - ' ' 2026-03-09 19:10:03.726917 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.726921 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.726926 | controller | | base64 -d)' 2026-03-09 19:10:03.726930 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.726936 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.726940 | controller | - ' exit 1' 2026-03-09 19:10:03.726945 | controller | - ' fi' 2026-03-09 19:10:03.726949 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.726953 | controller | - ' ' 2026-03-09 19:10:03.726957 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.726962 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.726966 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.726970 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.726981 | controller | --timeout=300s' 2026-03-09 19:10:03.726986 | controller | - ' ' 2026-03-09 19:10:03.726990 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.726994 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.727001 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.727007 | controller | - ' ' 2026-03-09 19:10:03.727013 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727019 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727025 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727031 | controller | - ' ' 2026-03-09 19:10:03.727037 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.727042 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.727048 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.727054 | controller | available"' 2026-03-09 19:10:03.727059 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.727065 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.727071 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.727077 | controller | - ' exit 1' 2026-03-09 19:10:03.727082 | controller | - ' fi' 2026-03-09 19:10:03.727092 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.727097 | controller | - ' ' 2026-03-09 19:10:03.727103 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.727108 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.727114 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727119 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.727125 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727130 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.727136 | controller | -- \' 2026-03-09 19:10:03.727141 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727159 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.727167 | controller | - ' logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.727171 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.727175 | controller | - ' logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.727179 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.727184 | controller | found' 2026-03-09 19:10:03.727190 | controller | - ' logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.727195 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.727199 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.727203 | controller | - ' ' 2026-03-09 19:10:03.727208 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.727212 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.727216 | controller | - ' ' 2026-03-09 19:10:03.727220 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.727225 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.727229 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.727236 | controller | - ' exit 1' 2026-03-09 19:10:03.727240 | controller | - ' fi' 2026-03-09 19:10:03.727244 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.727249 | controller | - ' ' 2026-03-09 19:10:03.727253 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.727257 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.727261 | controller | | base64 -d)' 2026-03-09 19:10:03.727266 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.727272 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.727277 | controller | - ' exit 1' 2026-03-09 19:10:03.727283 | controller | - ' fi' 2026-03-09 19:10:03.727289 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.727295 | controller | - ' ' 2026-03-09 19:10:03.727301 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.727307 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.727312 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.727317 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.727321 | controller | --timeout=300s' 2026-03-09 19:10:03.727325 | controller | - ' ' 2026-03-09 19:10:03.727339 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.727344 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.727348 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.727353 | controller | - ' ' 2026-03-09 19:10:03.727357 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727361 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727366 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727370 | controller | - ' ' 2026-03-09 19:10:03.727374 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.727378 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.727383 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.727387 | controller | available"' 2026-03-09 19:10:03.727391 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.727396 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.727400 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.727404 | controller | - ' exit 1' 2026-03-09 19:10:03.727408 | controller | - ' fi' 2026-03-09 19:10:03.727413 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.727417 | controller | - ' ' 2026-03-09 19:10:03.727421 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.727425 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.727430 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727434 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.727438 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727445 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.727449 | controller | -- \' 2026-03-09 19:10:03.727453 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727457 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.727462 | controller | - ' logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.727466 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.727470 | controller | - ' logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.727475 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.727479 | controller | found' 2026-03-09 19:10:03.727483 | controller | - ' logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.727487 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.727492 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.727496 | controller | - ' ' 2026-03-09 19:10:03.727500 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.727505 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.727509 | controller | - ' ' 2026-03-09 19:10:03.727513 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.727517 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.727522 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.727526 | controller | - ' exit 1' 2026-03-09 19:10:03.727530 | controller | - ' fi' 2026-03-09 19:10:03.727534 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.727539 | controller | - ' ' 2026-03-09 19:10:03.727543 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.727547 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.727551 | controller | | base64 -d)' 2026-03-09 19:10:03.727556 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.727560 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.727564 | controller | - ' exit 1' 2026-03-09 19:10:03.727569 | controller | - ' fi' 2026-03-09 19:10:03.727573 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.727577 | controller | - ' ' 2026-03-09 19:10:03.727582 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.727586 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.727590 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.727594 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.727599 | controller | --timeout=300s' 2026-03-09 19:10:03.727603 | controller | - ' ' 2026-03-09 19:10:03.727614 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.727618 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.727623 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.727627 | controller | - ' ' 2026-03-09 19:10:03.727631 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727635 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727640 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727647 | controller | - ' ' 2026-03-09 19:10:03.727651 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.727656 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.727660 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.727664 | controller | available"' 2026-03-09 19:10:03.727669 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.727673 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.727677 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.727681 | controller | - ' exit 1' 2026-03-09 19:10:03.727686 | controller | - ' fi' 2026-03-09 19:10:03.727690 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.727694 | controller | - ' ' 2026-03-09 19:10:03.727714 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.727720 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.727724 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727729 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.727735 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727740 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.727744 | controller | -- \' 2026-03-09 19:10:03.727749 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.727753 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.727757 | controller | - ' logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.727761 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.727766 | controller | - ' logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.727770 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.727774 | controller | found' 2026-03-09 19:10:03.727779 | controller | - ' logger.go:42: 18:54:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.727783 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.727787 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.727791 | controller | - ' ' 2026-03-09 19:10:03.727797 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.727801 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.727806 | controller | - ' ' 2026-03-09 19:10:03.727810 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.727814 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.727819 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.727823 | controller | - ' exit 1' 2026-03-09 19:10:03.727827 | controller | - ' fi' 2026-03-09 19:10:03.727831 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.727836 | controller | - ' ' 2026-03-09 19:10:03.727840 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.727844 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.727848 | controller | | base64 -d)' 2026-03-09 19:10:03.727853 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.727859 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.727863 | controller | - ' exit 1' 2026-03-09 19:10:03.727867 | controller | - ' fi' 2026-03-09 19:10:03.727872 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.727876 | controller | - ' ' 2026-03-09 19:10:03.727880 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.727885 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.727889 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.727893 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.727897 | controller | --timeout=300s' 2026-03-09 19:10:03.727902 | controller | - ' ' 2026-03-09 19:10:03.727906 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.727918 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.727923 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.727927 | controller | - ' ' 2026-03-09 19:10:03.727931 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727936 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727940 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.727944 | controller | - ' ' 2026-03-09 19:10:03.727949 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.727953 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.727957 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.727961 | controller | available"' 2026-03-09 19:10:03.727966 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.727970 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.727974 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.727978 | controller | - ' exit 1' 2026-03-09 19:10:03.727983 | controller | - ' fi' 2026-03-09 19:10:03.727987 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.727991 | controller | - ' ' 2026-03-09 19:10:03.727996 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.728000 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.728004 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728008 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.728013 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728017 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.728021 | controller | -- \' 2026-03-09 19:10:03.728025 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728030 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.728034 | controller | - ' logger.go:42: 18:54:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.728038 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.728045 | controller | - ' logger.go:42: 18:54:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.728049 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.728054 | controller | found' 2026-03-09 19:10:03.728058 | controller | - ' logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.728062 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.728066 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.728071 | controller | - ' ' 2026-03-09 19:10:03.728075 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.728079 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.728083 | controller | - ' ' 2026-03-09 19:10:03.728088 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.728092 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.728096 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.728101 | controller | - ' exit 1' 2026-03-09 19:10:03.728105 | controller | - ' fi' 2026-03-09 19:10:03.728110 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.728116 | controller | - ' ' 2026-03-09 19:10:03.728124 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.728128 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.728134 | controller | | base64 -d)' 2026-03-09 19:10:03.728139 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.728143 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.728168 | controller | - ' exit 1' 2026-03-09 19:10:03.728172 | controller | - ' fi' 2026-03-09 19:10:03.728176 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.728181 | controller | - ' ' 2026-03-09 19:10:03.728185 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.728189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.728194 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.728198 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.728202 | controller | --timeout=300s' 2026-03-09 19:10:03.728206 | controller | - ' ' 2026-03-09 19:10:03.728213 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.728225 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.728229 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.728233 | controller | - ' ' 2026-03-09 19:10:03.728238 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728242 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728246 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728251 | controller | - ' ' 2026-03-09 19:10:03.728255 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.728259 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.728264 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.728268 | controller | available"' 2026-03-09 19:10:03.728272 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.728279 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.728283 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.728287 | controller | - ' exit 1' 2026-03-09 19:10:03.728292 | controller | - ' fi' 2026-03-09 19:10:03.728296 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.728300 | controller | - ' ' 2026-03-09 19:10:03.728304 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.728309 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.728313 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728317 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.728322 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728326 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.728330 | controller | -- \' 2026-03-09 19:10:03.728334 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728339 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.728344 | controller | - ' logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.728349 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.728353 | controller | - ' logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.728357 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.728362 | controller | found' 2026-03-09 19:10:03.728366 | controller | - ' logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.728370 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.728375 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.728379 | controller | - ' ' 2026-03-09 19:10:03.728383 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.728389 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.728393 | controller | - ' ' 2026-03-09 19:10:03.728397 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.728402 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.728406 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.728410 | controller | - ' exit 1' 2026-03-09 19:10:03.728414 | controller | - ' fi' 2026-03-09 19:10:03.728419 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.728423 | controller | - ' ' 2026-03-09 19:10:03.728427 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.728432 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.728436 | controller | | base64 -d)' 2026-03-09 19:10:03.728440 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.728444 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.728449 | controller | - ' exit 1' 2026-03-09 19:10:03.728453 | controller | - ' fi' 2026-03-09 19:10:03.728457 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.728461 | controller | - ' ' 2026-03-09 19:10:03.728466 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.728470 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.728476 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.728480 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.728484 | controller | --timeout=300s' 2026-03-09 19:10:03.728490 | controller | - ' ' 2026-03-09 19:10:03.728494 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.728499 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.728510 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.728514 | controller | - ' ' 2026-03-09 19:10:03.728519 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728523 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728528 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728532 | controller | - ' ' 2026-03-09 19:10:03.728536 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.728540 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.728545 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.728549 | controller | available"' 2026-03-09 19:10:03.728553 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.728557 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.728562 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.728566 | controller | - ' exit 1' 2026-03-09 19:10:03.728570 | controller | - ' fi' 2026-03-09 19:10:03.728574 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.728579 | controller | - ' ' 2026-03-09 19:10:03.728583 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.728587 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.728591 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728596 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.728600 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728604 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.728608 | controller | -- \' 2026-03-09 19:10:03.728613 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728617 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.728621 | controller | - ' logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.728625 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.728630 | controller | - ' logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.728634 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.728639 | controller | found' 2026-03-09 19:10:03.728643 | controller | - ' logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.728647 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.728652 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.728658 | controller | - ' ' 2026-03-09 19:10:03.728664 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.728669 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.728673 | controller | - ' ' 2026-03-09 19:10:03.728677 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.728681 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.728686 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.728690 | controller | - ' exit 1' 2026-03-09 19:10:03.728694 | controller | - ' fi' 2026-03-09 19:10:03.728699 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.728703 | controller | - ' ' 2026-03-09 19:10:03.728707 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.728711 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.728716 | controller | | base64 -d)' 2026-03-09 19:10:03.728720 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.728724 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.728728 | controller | - ' exit 1' 2026-03-09 19:10:03.728733 | controller | - ' fi' 2026-03-09 19:10:03.728737 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.728741 | controller | - ' ' 2026-03-09 19:10:03.728745 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.728750 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.728754 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.728758 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.728762 | controller | --timeout=300s' 2026-03-09 19:10:03.728767 | controller | - ' ' 2026-03-09 19:10:03.728771 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.728775 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.728786 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.728791 | controller | - ' ' 2026-03-09 19:10:03.728795 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728802 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728807 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.728811 | controller | - ' ' 2026-03-09 19:10:03.728815 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.728820 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.728824 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.728828 | controller | available"' 2026-03-09 19:10:03.728833 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.728837 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.728841 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.728845 | controller | - ' exit 1' 2026-03-09 19:10:03.728850 | controller | - ' fi' 2026-03-09 19:10:03.728854 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.728858 | controller | - ' ' 2026-03-09 19:10:03.728862 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.728868 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.728873 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728877 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.728881 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728886 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.728890 | controller | -- \' 2026-03-09 19:10:03.728894 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.728899 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.728903 | controller | - ' logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.728907 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.728911 | controller | - ' logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.728916 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.728920 | controller | found' 2026-03-09 19:10:03.728924 | controller | - ' logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.728928 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.728933 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.728937 | controller | - ' ' 2026-03-09 19:10:03.728941 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.728946 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.728950 | controller | - ' ' 2026-03-09 19:10:03.728956 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.728960 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.728964 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.728969 | controller | - ' exit 1' 2026-03-09 19:10:03.728973 | controller | - ' fi' 2026-03-09 19:10:03.728977 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.728981 | controller | - ' ' 2026-03-09 19:10:03.728986 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.728990 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.728994 | controller | | base64 -d)' 2026-03-09 19:10:03.728998 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.729003 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.729007 | controller | - ' exit 1' 2026-03-09 19:10:03.729011 | controller | - ' fi' 2026-03-09 19:10:03.729016 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.729020 | controller | - ' ' 2026-03-09 19:10:03.729024 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.729028 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.729033 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.729037 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.729041 | controller | --timeout=300s' 2026-03-09 19:10:03.729045 | controller | - ' ' 2026-03-09 19:10:03.729050 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.729054 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.729066 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.729071 | controller | - ' ' 2026-03-09 19:10:03.729075 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729080 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729084 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729088 | controller | - ' ' 2026-03-09 19:10:03.729093 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.729097 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.729101 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.729105 | controller | available"' 2026-03-09 19:10:03.729110 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.729114 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.729118 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.729122 | controller | - ' exit 1' 2026-03-09 19:10:03.729127 | controller | - ' fi' 2026-03-09 19:10:03.729131 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.729135 | controller | - ' ' 2026-03-09 19:10:03.729140 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.729144 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.729162 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729167 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.729171 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729175 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.729179 | controller | -- \' 2026-03-09 19:10:03.729184 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729188 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.729192 | controller | - ' logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.729197 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.729201 | controller | - ' logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.729205 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.729209 | controller | found' 2026-03-09 19:10:03.729214 | controller | - ' logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.729218 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.729222 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.729227 | controller | - ' ' 2026-03-09 19:10:03.729231 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.729235 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.729239 | controller | - ' ' 2026-03-09 19:10:03.729244 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.729248 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.729252 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.729257 | controller | - ' exit 1' 2026-03-09 19:10:03.729263 | controller | - ' fi' 2026-03-09 19:10:03.729267 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.729272 | controller | - ' ' 2026-03-09 19:10:03.729276 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.729280 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.729285 | controller | | base64 -d)' 2026-03-09 19:10:03.729291 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.729295 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.729299 | controller | - ' exit 1' 2026-03-09 19:10:03.729304 | controller | - ' fi' 2026-03-09 19:10:03.729308 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.729312 | controller | - ' ' 2026-03-09 19:10:03.729317 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.729321 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.729325 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.729329 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.729334 | controller | --timeout=300s' 2026-03-09 19:10:03.729338 | controller | - ' ' 2026-03-09 19:10:03.729342 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.729347 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.729351 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.729355 | controller | - ' ' 2026-03-09 19:10:03.729367 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729371 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729376 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729380 | controller | - ' ' 2026-03-09 19:10:03.729384 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.729389 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.729393 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.729397 | controller | available"' 2026-03-09 19:10:03.729401 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.729407 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.729412 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.729416 | controller | - ' exit 1' 2026-03-09 19:10:03.729420 | controller | - ' fi' 2026-03-09 19:10:03.729425 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.729429 | controller | - ' ' 2026-03-09 19:10:03.729433 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.729437 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.729442 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729446 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.729450 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729455 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.729461 | controller | -- \' 2026-03-09 19:10:03.729465 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729469 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.729474 | controller | - ' logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.729478 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.729482 | controller | - ' logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.729487 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.729491 | controller | found' 2026-03-09 19:10:03.729495 | controller | - ' logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.729499 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.729504 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.729508 | controller | - ' ' 2026-03-09 19:10:03.729512 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.729516 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.729521 | controller | - ' ' 2026-03-09 19:10:03.729525 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.729529 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.729534 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.729538 | controller | - ' exit 1' 2026-03-09 19:10:03.729542 | controller | - ' fi' 2026-03-09 19:10:03.729546 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.729552 | controller | - ' ' 2026-03-09 19:10:03.729557 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.729561 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.729565 | controller | | base64 -d)' 2026-03-09 19:10:03.729570 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.729574 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.729578 | controller | - ' exit 1' 2026-03-09 19:10:03.729582 | controller | - ' fi' 2026-03-09 19:10:03.729587 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.729591 | controller | - ' ' 2026-03-09 19:10:03.729595 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.729600 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.729604 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.729608 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.729613 | controller | --timeout=300s' 2026-03-09 19:10:03.729617 | controller | - ' ' 2026-03-09 19:10:03.729621 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.729625 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.729630 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.729649 | controller | - ' ' 2026-03-09 19:10:03.729659 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729664 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729668 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729674 | controller | - ' ' 2026-03-09 19:10:03.729679 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.729683 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.729687 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.729692 | controller | available"' 2026-03-09 19:10:03.729696 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.729700 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.729705 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.729709 | controller | - ' exit 1' 2026-03-09 19:10:03.729713 | controller | - ' fi' 2026-03-09 19:10:03.729718 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.729722 | controller | - ' ' 2026-03-09 19:10:03.729726 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.729730 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.729735 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729739 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.729743 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729748 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.729754 | controller | -- \' 2026-03-09 19:10:03.729758 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.729762 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.729767 | controller | - ' logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.729773 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.729779 | controller | - ' logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.729783 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.729787 | controller | found' 2026-03-09 19:10:03.729792 | controller | - ' logger.go:42: 18:54:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.729796 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.729800 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.729805 | controller | - ' ' 2026-03-09 19:10:03.729809 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.729813 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.729817 | controller | - ' ' 2026-03-09 19:10:03.729822 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.729826 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.729830 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.729835 | controller | - ' exit 1' 2026-03-09 19:10:03.729839 | controller | - ' fi' 2026-03-09 19:10:03.729843 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.729848 | controller | - ' ' 2026-03-09 19:10:03.729852 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.729856 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.729860 | controller | | base64 -d)' 2026-03-09 19:10:03.729865 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.729869 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.729875 | controller | - ' exit 1' 2026-03-09 19:10:03.729879 | controller | - ' fi' 2026-03-09 19:10:03.729884 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.729888 | controller | - ' ' 2026-03-09 19:10:03.729892 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.729897 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.729901 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.729905 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.729909 | controller | --timeout=300s' 2026-03-09 19:10:03.729914 | controller | - ' ' 2026-03-09 19:10:03.729918 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.729922 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.729926 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.729931 | controller | - ' ' 2026-03-09 19:10:03.729935 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729946 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729951 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.729955 | controller | - ' ' 2026-03-09 19:10:03.729959 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.729964 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.729968 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.729972 | controller | available"' 2026-03-09 19:10:03.729977 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.729981 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.729985 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.729990 | controller | - ' exit 1' 2026-03-09 19:10:03.729994 | controller | - ' fi' 2026-03-09 19:10:03.729998 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.730002 | controller | - ' ' 2026-03-09 19:10:03.730007 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.730011 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.730015 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730019 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.730024 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730028 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.730032 | controller | -- \' 2026-03-09 19:10:03.730036 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730041 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.730045 | controller | - ' logger.go:42: 18:54:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.730049 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.730053 | controller | - ' logger.go:42: 18:54:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.730059 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.730064 | controller | found' 2026-03-09 19:10:03.730068 | controller | - ' logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.730072 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.730077 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.730081 | controller | - ' ' 2026-03-09 19:10:03.730085 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.730089 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.730094 | controller | - ' ' 2026-03-09 19:10:03.730098 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.730102 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.730106 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.730111 | controller | - ' exit 1' 2026-03-09 19:10:03.730115 | controller | - ' fi' 2026-03-09 19:10:03.730119 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.730124 | controller | - ' ' 2026-03-09 19:10:03.730128 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.730132 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.730136 | controller | | base64 -d)' 2026-03-09 19:10:03.730141 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.730161 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.730168 | controller | - ' exit 1' 2026-03-09 19:10:03.730172 | controller | - ' fi' 2026-03-09 19:10:03.730176 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.730181 | controller | - ' ' 2026-03-09 19:10:03.730185 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.730189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.730196 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.730201 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.730205 | controller | --timeout=300s' 2026-03-09 19:10:03.730209 | controller | - ' ' 2026-03-09 19:10:03.730214 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.730218 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.730222 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.730227 | controller | - ' ' 2026-03-09 19:10:03.730231 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730243 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730247 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730251 | controller | - ' ' 2026-03-09 19:10:03.730256 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.730260 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.730264 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.730269 | controller | available"' 2026-03-09 19:10:03.730273 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.730277 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.730285 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.730289 | controller | - ' exit 1' 2026-03-09 19:10:03.730294 | controller | - ' fi' 2026-03-09 19:10:03.730298 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.730302 | controller | - ' ' 2026-03-09 19:10:03.730307 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.730311 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.730315 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730319 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.730324 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730328 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.730332 | controller | -- \' 2026-03-09 19:10:03.730337 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730341 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.730345 | controller | - ' logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.730349 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.730354 | controller | - ' logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.730358 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.730362 | controller | found' 2026-03-09 19:10:03.730367 | controller | - ' logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.730371 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.730375 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.730379 | controller | - ' ' 2026-03-09 19:10:03.730385 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.730390 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.730394 | controller | - ' ' 2026-03-09 19:10:03.730398 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.730403 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.730407 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.730411 | controller | - ' exit 1' 2026-03-09 19:10:03.730415 | controller | - ' fi' 2026-03-09 19:10:03.730420 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.730424 | controller | - ' ' 2026-03-09 19:10:03.730428 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.730433 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.730437 | controller | | base64 -d)' 2026-03-09 19:10:03.730441 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.730445 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.730450 | controller | - ' exit 1' 2026-03-09 19:10:03.730454 | controller | - ' fi' 2026-03-09 19:10:03.730458 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.730462 | controller | - ' ' 2026-03-09 19:10:03.730467 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.730471 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.730475 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.730481 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.730486 | controller | --timeout=300s' 2026-03-09 19:10:03.730490 | controller | - ' ' 2026-03-09 19:10:03.730494 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.730500 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.730504 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.730509 | controller | - ' ' 2026-03-09 19:10:03.730513 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730517 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730528 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730533 | controller | - ' ' 2026-03-09 19:10:03.730537 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.730541 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.730546 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.730550 | controller | available"' 2026-03-09 19:10:03.730554 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.730559 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.730563 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.730567 | controller | - ' exit 1' 2026-03-09 19:10:03.730571 | controller | - ' fi' 2026-03-09 19:10:03.730576 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.730580 | controller | - ' ' 2026-03-09 19:10:03.730584 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.730588 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.730593 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730597 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.730601 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730605 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.730610 | controller | -- \' 2026-03-09 19:10:03.730614 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730618 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.730623 | controller | - ' logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.730627 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.730631 | controller | - ' logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.730635 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.730640 | controller | found' 2026-03-09 19:10:03.730644 | controller | - ' logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.730648 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.730652 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.730657 | controller | - ' ' 2026-03-09 19:10:03.730661 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.730667 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.730671 | controller | - ' ' 2026-03-09 19:10:03.730676 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.730680 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.730684 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.730689 | controller | - ' exit 1' 2026-03-09 19:10:03.730693 | controller | - ' fi' 2026-03-09 19:10:03.730697 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.730701 | controller | - ' ' 2026-03-09 19:10:03.730707 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.730712 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.730716 | controller | | base64 -d)' 2026-03-09 19:10:03.730720 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.730725 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.730729 | controller | - ' exit 1' 2026-03-09 19:10:03.730733 | controller | - ' fi' 2026-03-09 19:10:03.730737 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.730742 | controller | - ' ' 2026-03-09 19:10:03.730746 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.730750 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.730754 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.730760 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.730765 | controller | --timeout=300s' 2026-03-09 19:10:03.730769 | controller | - ' ' 2026-03-09 19:10:03.730773 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.730777 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.730782 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.730786 | controller | - ' ' 2026-03-09 19:10:03.730790 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730795 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730808 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.730813 | controller | - ' ' 2026-03-09 19:10:03.730817 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.730821 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.730826 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.730830 | controller | available"' 2026-03-09 19:10:03.730836 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.730841 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.730846 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.730850 | controller | - ' exit 1' 2026-03-09 19:10:03.730854 | controller | - ' fi' 2026-03-09 19:10:03.730859 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.730863 | controller | - ' ' 2026-03-09 19:10:03.730867 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.730871 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.730878 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730882 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.730887 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730891 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.730895 | controller | -- \' 2026-03-09 19:10:03.730900 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.730904 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.730908 | controller | - ' logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.730912 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.730917 | controller | - ' logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.730921 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.730925 | controller | found' 2026-03-09 19:10:03.730930 | controller | - ' logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.730934 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.730938 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.730942 | controller | - ' ' 2026-03-09 19:10:03.730947 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.730951 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.730955 | controller | - ' ' 2026-03-09 19:10:03.730959 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.730964 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.730968 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.730972 | controller | - ' exit 1' 2026-03-09 19:10:03.730977 | controller | - ' fi' 2026-03-09 19:10:03.730981 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.730985 | controller | - ' ' 2026-03-09 19:10:03.730990 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.730994 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.730998 | controller | | base64 -d)' 2026-03-09 19:10:03.731002 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.731008 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.731013 | controller | - ' exit 1' 2026-03-09 19:10:03.731017 | controller | - ' fi' 2026-03-09 19:10:03.731021 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.731025 | controller | - ' ' 2026-03-09 19:10:03.731030 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.731034 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.731038 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.731042 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.731047 | controller | --timeout=300s' 2026-03-09 19:10:03.731051 | controller | - ' ' 2026-03-09 19:10:03.731055 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.731060 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.731064 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.731070 | controller | - ' ' 2026-03-09 19:10:03.731074 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731078 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731082 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731094 | controller | - ' ' 2026-03-09 19:10:03.731099 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.731104 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.731109 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.731114 | controller | available"' 2026-03-09 19:10:03.731120 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.731127 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.731133 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.731137 | controller | - ' exit 1' 2026-03-09 19:10:03.731141 | controller | - ' fi' 2026-03-09 19:10:03.731156 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.731163 | controller | - ' ' 2026-03-09 19:10:03.731167 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.731171 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.731176 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731180 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.731184 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731188 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.731193 | controller | -- \' 2026-03-09 19:10:03.731197 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731201 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.731206 | controller | - ' logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.731210 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.731214 | controller | - ' logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.731219 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.731223 | controller | found' 2026-03-09 19:10:03.731227 | controller | - ' logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.731231 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.731236 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.731240 | controller | - ' ' 2026-03-09 19:10:03.731244 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.731249 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.731253 | controller | - ' ' 2026-03-09 19:10:03.731257 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.731261 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.731266 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.731270 | controller | - ' exit 1' 2026-03-09 19:10:03.731274 | controller | - ' fi' 2026-03-09 19:10:03.731281 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.731285 | controller | - ' ' 2026-03-09 19:10:03.731289 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.731293 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.731298 | controller | | base64 -d)' 2026-03-09 19:10:03.731302 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.731306 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.731311 | controller | - ' exit 1' 2026-03-09 19:10:03.731315 | controller | - ' fi' 2026-03-09 19:10:03.731319 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.731323 | controller | - ' ' 2026-03-09 19:10:03.731328 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.731333 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.731338 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.731343 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.731348 | controller | --timeout=300s' 2026-03-09 19:10:03.731352 | controller | - ' ' 2026-03-09 19:10:03.731356 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.731360 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.731365 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.731369 | controller | - ' ' 2026-03-09 19:10:03.731373 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731378 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731382 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731386 | controller | - ' ' 2026-03-09 19:10:03.731400 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.731406 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.731411 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.731415 | controller | available"' 2026-03-09 19:10:03.731419 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.731425 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.731431 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.731435 | controller | - ' exit 1' 2026-03-09 19:10:03.731439 | controller | - ' fi' 2026-03-09 19:10:03.731443 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.731448 | controller | - ' ' 2026-03-09 19:10:03.731452 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.731456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.731460 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731465 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.731469 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731473 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.731479 | controller | -- \' 2026-03-09 19:10:03.731484 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731488 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.731492 | controller | - ' logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.731496 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.731501 | controller | - ' logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.731505 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.731509 | controller | found' 2026-03-09 19:10:03.731514 | controller | - ' logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.731518 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.731522 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.731527 | controller | - ' ' 2026-03-09 19:10:03.731531 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.731535 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.731539 | controller | - ' ' 2026-03-09 19:10:03.731544 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.731548 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.731552 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.731556 | controller | - ' exit 1' 2026-03-09 19:10:03.731561 | controller | - ' fi' 2026-03-09 19:10:03.731565 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.731569 | controller | - ' ' 2026-03-09 19:10:03.731574 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.731578 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.731582 | controller | | base64 -d)' 2026-03-09 19:10:03.731586 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.731591 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.731595 | controller | - ' exit 1' 2026-03-09 19:10:03.731599 | controller | - ' fi' 2026-03-09 19:10:03.731603 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.731608 | controller | - ' ' 2026-03-09 19:10:03.731612 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.731616 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.731620 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.731625 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.731629 | controller | --timeout=300s' 2026-03-09 19:10:03.731633 | controller | - ' ' 2026-03-09 19:10:03.731638 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.731642 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.731646 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.731650 | controller | - ' ' 2026-03-09 19:10:03.731655 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731659 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731663 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731667 | controller | - ' ' 2026-03-09 19:10:03.731674 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.731686 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.731691 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.731715 | controller | available"' 2026-03-09 19:10:03.731721 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.731725 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.731729 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.731734 | controller | - ' exit 1' 2026-03-09 19:10:03.731738 | controller | - ' fi' 2026-03-09 19:10:03.731742 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.731746 | controller | - ' ' 2026-03-09 19:10:03.731751 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.731755 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.731759 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731763 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.731768 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731772 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.731776 | controller | -- \' 2026-03-09 19:10:03.731781 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.731785 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.731791 | controller | - ' logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.731796 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.731800 | controller | - ' logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.731804 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.731809 | controller | found' 2026-03-09 19:10:03.731813 | controller | - ' logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.731817 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.731822 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.731826 | controller | - ' ' 2026-03-09 19:10:03.731830 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.731834 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.731839 | controller | - ' ' 2026-03-09 19:10:03.731843 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.731847 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.731852 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.731856 | controller | - ' exit 1' 2026-03-09 19:10:03.731860 | controller | - ' fi' 2026-03-09 19:10:03.731865 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.731869 | controller | - ' ' 2026-03-09 19:10:03.731873 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.731877 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.731882 | controller | | base64 -d)' 2026-03-09 19:10:03.731886 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.731890 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.731896 | controller | - ' exit 1' 2026-03-09 19:10:03.731901 | controller | - ' fi' 2026-03-09 19:10:03.731905 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.731909 | controller | - ' ' 2026-03-09 19:10:03.731913 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.731918 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.731922 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.731926 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.731931 | controller | --timeout=300s' 2026-03-09 19:10:03.731935 | controller | - ' ' 2026-03-09 19:10:03.731939 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.731943 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.731948 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.731952 | controller | - ' ' 2026-03-09 19:10:03.731956 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731961 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731966 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.731972 | controller | - ' ' 2026-03-09 19:10:03.731979 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.731996 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.732004 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.732010 | controller | available"' 2026-03-09 19:10:03.732016 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.732024 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.732030 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.732036 | controller | - ' exit 1' 2026-03-09 19:10:03.732042 | controller | - ' fi' 2026-03-09 19:10:03.732048 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.732054 | controller | - ' ' 2026-03-09 19:10:03.732060 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.732065 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.732071 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732077 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.732082 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732088 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.732094 | controller | -- \' 2026-03-09 19:10:03.732099 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732105 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.732110 | controller | - ' logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.732116 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.732122 | controller | - ' logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.732131 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.732136 | controller | found' 2026-03-09 19:10:03.732142 | controller | - ' logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.732168 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.732173 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.732182 | controller | - ' ' 2026-03-09 19:10:03.732188 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.732194 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.732200 | controller | - ' ' 2026-03-09 19:10:03.732205 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.732210 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.732214 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.732218 | controller | - ' exit 1' 2026-03-09 19:10:03.732223 | controller | - ' fi' 2026-03-09 19:10:03.732227 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.732231 | controller | - ' ' 2026-03-09 19:10:03.732236 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.732240 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.732244 | controller | | base64 -d)' 2026-03-09 19:10:03.732249 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.732253 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.732257 | controller | - ' exit 1' 2026-03-09 19:10:03.732261 | controller | - ' fi' 2026-03-09 19:10:03.732266 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.732270 | controller | - ' ' 2026-03-09 19:10:03.732274 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.732278 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.732283 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.732287 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.732291 | controller | --timeout=300s' 2026-03-09 19:10:03.732296 | controller | - ' ' 2026-03-09 19:10:03.732300 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.732304 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.732308 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.732313 | controller | - ' ' 2026-03-09 19:10:03.732317 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732321 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732326 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732330 | controller | - ' ' 2026-03-09 19:10:03.732334 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.732338 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.732352 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.732357 | controller | available"' 2026-03-09 19:10:03.732361 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.732366 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.732373 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.732377 | controller | - ' exit 1' 2026-03-09 19:10:03.732381 | controller | - ' fi' 2026-03-09 19:10:03.732386 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.732390 | controller | - ' ' 2026-03-09 19:10:03.732394 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.732399 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.732403 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732407 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.732411 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732416 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.732420 | controller | -- \' 2026-03-09 19:10:03.732424 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732429 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.732435 | controller | - ' logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.732440 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.732444 | controller | - ' logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.732448 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.732453 | controller | found' 2026-03-09 19:10:03.732457 | controller | - ' logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.732461 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.732466 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.732470 | controller | - ' ' 2026-03-09 19:10:03.732474 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.732478 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.732483 | controller | - ' ' 2026-03-09 19:10:03.732487 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.732491 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.732496 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.732500 | controller | - ' exit 1' 2026-03-09 19:10:03.732504 | controller | - ' fi' 2026-03-09 19:10:03.732509 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.732514 | controller | - ' ' 2026-03-09 19:10:03.732518 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.732522 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.732527 | controller | | base64 -d)' 2026-03-09 19:10:03.732531 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.732535 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.732539 | controller | - ' exit 1' 2026-03-09 19:10:03.732544 | controller | - ' fi' 2026-03-09 19:10:03.732550 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.732554 | controller | - ' ' 2026-03-09 19:10:03.732559 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.732563 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.732567 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.732573 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.732577 | controller | --timeout=300s' 2026-03-09 19:10:03.732583 | controller | - ' ' 2026-03-09 19:10:03.732588 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.732592 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.732596 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.732600 | controller | - ' ' 2026-03-09 19:10:03.732605 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732609 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732613 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732617 | controller | - ' ' 2026-03-09 19:10:03.732622 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.732626 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.732630 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.732641 | controller | available"' 2026-03-09 19:10:03.732646 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.732650 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.732655 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.732659 | controller | - ' exit 1' 2026-03-09 19:10:03.732663 | controller | - ' fi' 2026-03-09 19:10:03.732668 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.732672 | controller | - ' ' 2026-03-09 19:10:03.732676 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.732680 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.732685 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732689 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.732693 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732697 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.732702 | controller | -- \' 2026-03-09 19:10:03.732706 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732710 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.732714 | controller | - ' logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.732719 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.732727 | controller | - ' logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.732732 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.732737 | controller | found' 2026-03-09 19:10:03.732741 | controller | - ' logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.732745 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.732749 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.732753 | controller | - ' ' 2026-03-09 19:10:03.732758 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.732762 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.732768 | controller | - ' ' 2026-03-09 19:10:03.732772 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.732776 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.732781 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.732785 | controller | - ' exit 1' 2026-03-09 19:10:03.732789 | controller | - ' fi' 2026-03-09 19:10:03.732794 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.732798 | controller | - ' ' 2026-03-09 19:10:03.732802 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.732806 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.732811 | controller | | base64 -d)' 2026-03-09 19:10:03.732815 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.732819 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.732823 | controller | - ' exit 1' 2026-03-09 19:10:03.732828 | controller | - ' fi' 2026-03-09 19:10:03.732832 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.732836 | controller | - ' ' 2026-03-09 19:10:03.732840 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.732845 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.732849 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.732853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.732857 | controller | --timeout=300s' 2026-03-09 19:10:03.732862 | controller | - ' ' 2026-03-09 19:10:03.732866 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.732870 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.732874 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.732879 | controller | - ' ' 2026-03-09 19:10:03.732883 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732887 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732891 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.732896 | controller | - ' ' 2026-03-09 19:10:03.732900 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.732906 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.732910 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.732915 | controller | available"' 2026-03-09 19:10:03.732926 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.732930 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.732935 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.732939 | controller | - ' exit 1' 2026-03-09 19:10:03.732943 | controller | - ' fi' 2026-03-09 19:10:03.732951 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.732955 | controller | - ' ' 2026-03-09 19:10:03.732960 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.732964 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.732968 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732975 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.732979 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732983 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.732987 | controller | -- \' 2026-03-09 19:10:03.732992 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.732996 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.733000 | controller | - ' logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.733005 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.733011 | controller | - ' logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.733017 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.733023 | controller | found' 2026-03-09 19:10:03.733029 | controller | - ' logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.733035 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.733041 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.733046 | controller | - ' ' 2026-03-09 19:10:03.733052 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.733057 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.733063 | controller | - ' ' 2026-03-09 19:10:03.733068 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.733072 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.733078 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.733083 | controller | - ' exit 1' 2026-03-09 19:10:03.733088 | controller | - ' fi' 2026-03-09 19:10:03.733092 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.733097 | controller | - ' ' 2026-03-09 19:10:03.733101 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.733105 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.733109 | controller | | base64 -d)' 2026-03-09 19:10:03.733114 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.733118 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.733122 | controller | - ' exit 1' 2026-03-09 19:10:03.733126 | controller | - ' fi' 2026-03-09 19:10:03.733131 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.733135 | controller | - ' ' 2026-03-09 19:10:03.733139 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.733144 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.733164 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.733168 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.733173 | controller | --timeout=300s' 2026-03-09 19:10:03.733177 | controller | - ' ' 2026-03-09 19:10:03.733181 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.733186 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.733190 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.733197 | controller | - ' ' 2026-03-09 19:10:03.733202 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733206 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733210 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733214 | controller | - ' ' 2026-03-09 19:10:03.733218 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.733223 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.733228 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.733234 | controller | available"' 2026-03-09 19:10:03.733238 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.733254 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.733259 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.733263 | controller | - ' exit 1' 2026-03-09 19:10:03.733267 | controller | - ' fi' 2026-03-09 19:10:03.733271 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.733276 | controller | - ' ' 2026-03-09 19:10:03.733283 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.733287 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.733291 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733297 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.733302 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733306 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.733311 | controller | -- \' 2026-03-09 19:10:03.733315 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733319 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.733323 | controller | - ' logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.733328 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.733332 | controller | - ' logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.733336 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.733340 | controller | found' 2026-03-09 19:10:03.733345 | controller | - ' logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.733349 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.733353 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.733358 | controller | - ' ' 2026-03-09 19:10:03.733362 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.733366 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.733370 | controller | - ' ' 2026-03-09 19:10:03.733375 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.733379 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.733383 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.733387 | controller | - ' exit 1' 2026-03-09 19:10:03.733392 | controller | - ' fi' 2026-03-09 19:10:03.733396 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.733402 | controller | - ' ' 2026-03-09 19:10:03.733408 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.733412 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.733417 | controller | | base64 -d)' 2026-03-09 19:10:03.733421 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.733425 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.733430 | controller | - ' exit 1' 2026-03-09 19:10:03.733434 | controller | - ' fi' 2026-03-09 19:10:03.733438 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.733442 | controller | - ' ' 2026-03-09 19:10:03.733447 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.733451 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.733455 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.733459 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.733464 | controller | --timeout=300s' 2026-03-09 19:10:03.733468 | controller | - ' ' 2026-03-09 19:10:03.733472 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.733476 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.733481 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.733485 | controller | - ' ' 2026-03-09 19:10:03.733489 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733493 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733498 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733502 | controller | - ' ' 2026-03-09 19:10:03.733506 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.733510 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.733515 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.733519 | controller | available"' 2026-03-09 19:10:03.733523 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.733528 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.733541 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.733546 | controller | - ' exit 1' 2026-03-09 19:10:03.733550 | controller | - ' fi' 2026-03-09 19:10:03.733555 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.733559 | controller | - ' ' 2026-03-09 19:10:03.733563 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.733568 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.733572 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733576 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.733581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733585 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.733589 | controller | -- \' 2026-03-09 19:10:03.733594 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733600 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.733604 | controller | - ' logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.733608 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.733613 | controller | - ' logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.733617 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.733621 | controller | found' 2026-03-09 19:10:03.733625 | controller | - ' logger.go:42: 18:54:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.733630 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.733634 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.733638 | controller | - ' ' 2026-03-09 19:10:03.733643 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.733647 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.733651 | controller | - ' ' 2026-03-09 19:10:03.733655 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.733660 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.733664 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.733668 | controller | - ' exit 1' 2026-03-09 19:10:03.733675 | controller | - ' fi' 2026-03-09 19:10:03.733679 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.733683 | controller | - ' ' 2026-03-09 19:10:03.733688 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.733692 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.733696 | controller | | base64 -d)' 2026-03-09 19:10:03.733701 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.733705 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.733709 | controller | - ' exit 1' 2026-03-09 19:10:03.733713 | controller | - ' fi' 2026-03-09 19:10:03.733718 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.733722 | controller | - ' ' 2026-03-09 19:10:03.733726 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.733730 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.733735 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.733739 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.733743 | controller | --timeout=300s' 2026-03-09 19:10:03.733748 | controller | - ' ' 2026-03-09 19:10:03.733752 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.733756 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.733760 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.733765 | controller | - ' ' 2026-03-09 19:10:03.733770 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733775 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733779 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.733783 | controller | - ' ' 2026-03-09 19:10:03.733788 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.733794 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.733798 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.733802 | controller | available"' 2026-03-09 19:10:03.733807 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.733811 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.733826 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.733831 | controller | - ' exit 1' 2026-03-09 19:10:03.733835 | controller | - ' fi' 2026-03-09 19:10:03.733840 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.733844 | controller | - ' ' 2026-03-09 19:10:03.733848 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.733853 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.733857 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733863 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.733867 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733872 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.733876 | controller | -- \' 2026-03-09 19:10:03.733881 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.733885 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.733889 | controller | - ' logger.go:42: 18:54:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.733893 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.733898 | controller | - ' logger.go:42: 18:54:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.733902 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.733906 | controller | found' 2026-03-09 19:10:03.733911 | controller | - ' logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.733915 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.733919 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.733923 | controller | - ' ' 2026-03-09 19:10:03.733928 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.733932 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.733936 | controller | - ' ' 2026-03-09 19:10:03.733941 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.733945 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.733949 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.733953 | controller | - ' exit 1' 2026-03-09 19:10:03.733958 | controller | - ' fi' 2026-03-09 19:10:03.733962 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.733966 | controller | - ' ' 2026-03-09 19:10:03.733970 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.733975 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.733979 | controller | | base64 -d)' 2026-03-09 19:10:03.733983 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.733988 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.733992 | controller | - ' exit 1' 2026-03-09 19:10:03.733998 | controller | - ' fi' 2026-03-09 19:10:03.734002 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.734007 | controller | - ' ' 2026-03-09 19:10:03.734013 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.734019 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.734025 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.734031 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.734038 | controller | --timeout=300s' 2026-03-09 19:10:03.734042 | controller | - ' ' 2026-03-09 19:10:03.734047 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.734051 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.734058 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.734062 | controller | - ' ' 2026-03-09 19:10:03.734066 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734071 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734075 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734079 | controller | - ' ' 2026-03-09 19:10:03.734084 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.734088 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.734093 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.734097 | controller | available"' 2026-03-09 19:10:03.734102 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.734107 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.734112 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.734117 | controller | - ' exit 1' 2026-03-09 19:10:03.734131 | controller | - ' fi' 2026-03-09 19:10:03.734136 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.734140 | controller | - ' ' 2026-03-09 19:10:03.734161 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.734169 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.734173 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734177 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.734182 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734186 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.734190 | controller | -- \' 2026-03-09 19:10:03.734194 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734199 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.734203 | controller | - ' logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.734207 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.734211 | controller | - ' logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.734216 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.734222 | controller | found' 2026-03-09 19:10:03.734227 | controller | - ' logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.734231 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.734235 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.734240 | controller | - ' ' 2026-03-09 19:10:03.734244 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.734248 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.734252 | controller | - ' ' 2026-03-09 19:10:03.734257 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.734261 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.734265 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.734270 | controller | - ' exit 1' 2026-03-09 19:10:03.734274 | controller | - ' fi' 2026-03-09 19:10:03.734278 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.734282 | controller | - ' ' 2026-03-09 19:10:03.734287 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.734291 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.734295 | controller | | base64 -d)' 2026-03-09 19:10:03.734300 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.734304 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.734308 | controller | - ' exit 1' 2026-03-09 19:10:03.734312 | controller | - ' fi' 2026-03-09 19:10:03.734317 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.734321 | controller | - ' ' 2026-03-09 19:10:03.734325 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.734330 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.734334 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.734338 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.734342 | controller | --timeout=300s' 2026-03-09 19:10:03.734347 | controller | - ' ' 2026-03-09 19:10:03.734351 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.734355 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.734360 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.734364 | controller | - ' ' 2026-03-09 19:10:03.734368 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734372 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734377 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734381 | controller | - ' ' 2026-03-09 19:10:03.734385 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.734390 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.734394 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.734398 | controller | available"' 2026-03-09 19:10:03.734402 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.734407 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.734411 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.734417 | controller | - ' exit 1' 2026-03-09 19:10:03.734421 | controller | - ' fi' 2026-03-09 19:10:03.734433 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.734438 | controller | - ' ' 2026-03-09 19:10:03.734442 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.734446 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.734451 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734455 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.734459 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734464 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.734468 | controller | -- \' 2026-03-09 19:10:03.734472 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734481 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.734485 | controller | - ' logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.734489 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.734494 | controller | - ' logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.734498 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.734502 | controller | found' 2026-03-09 19:10:03.734506 | controller | - ' logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.734511 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.734515 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.734519 | controller | - ' ' 2026-03-09 19:10:03.734524 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.734528 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.734532 | controller | - ' ' 2026-03-09 19:10:03.734536 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.734541 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.734545 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.734549 | controller | - ' exit 1' 2026-03-09 19:10:03.734554 | controller | - ' fi' 2026-03-09 19:10:03.734558 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.734562 | controller | - ' ' 2026-03-09 19:10:03.734566 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.734571 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.734575 | controller | | base64 -d)' 2026-03-09 19:10:03.734579 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.734583 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.734588 | controller | - ' exit 1' 2026-03-09 19:10:03.734592 | controller | - ' fi' 2026-03-09 19:10:03.734596 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.734600 | controller | - ' ' 2026-03-09 19:10:03.734605 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.734609 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.734613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.734618 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.734623 | controller | --timeout=300s' 2026-03-09 19:10:03.734628 | controller | - ' ' 2026-03-09 19:10:03.734632 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.734636 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.734641 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.734645 | controller | - ' ' 2026-03-09 19:10:03.734649 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734654 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734658 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734662 | controller | - ' ' 2026-03-09 19:10:03.734666 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.734671 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.734675 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.734679 | controller | available"' 2026-03-09 19:10:03.734684 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.734690 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.734694 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.734698 | controller | - ' exit 1' 2026-03-09 19:10:03.734704 | controller | - ' fi' 2026-03-09 19:10:03.734708 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.734720 | controller | - ' ' 2026-03-09 19:10:03.734725 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.734729 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.734733 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734737 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.734742 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734746 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.734750 | controller | -- \' 2026-03-09 19:10:03.734755 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.734759 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.734763 | controller | - ' logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.734767 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.734772 | controller | - ' logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.734776 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.734780 | controller | found' 2026-03-09 19:10:03.734786 | controller | - ' logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.734790 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.734795 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.734799 | controller | - ' ' 2026-03-09 19:10:03.734803 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.734808 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.734814 | controller | - ' ' 2026-03-09 19:10:03.734818 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.734822 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.734827 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.734831 | controller | - ' exit 1' 2026-03-09 19:10:03.734835 | controller | - ' fi' 2026-03-09 19:10:03.734839 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.734844 | controller | - ' ' 2026-03-09 19:10:03.734848 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.734852 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.734856 | controller | | base64 -d)' 2026-03-09 19:10:03.734861 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.734865 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.734869 | controller | - ' exit 1' 2026-03-09 19:10:03.734874 | controller | - ' fi' 2026-03-09 19:10:03.734878 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.734882 | controller | - ' ' 2026-03-09 19:10:03.734887 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.734891 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.734895 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.734899 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.734904 | controller | --timeout=300s' 2026-03-09 19:10:03.734908 | controller | - ' ' 2026-03-09 19:10:03.734912 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.734917 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.734921 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.734925 | controller | - ' ' 2026-03-09 19:10:03.734929 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734934 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734938 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.734942 | controller | - ' ' 2026-03-09 19:10:03.734947 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.734951 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.734955 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.734959 | controller | available"' 2026-03-09 19:10:03.734964 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.734968 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.734972 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.734977 | controller | - ' exit 1' 2026-03-09 19:10:03.734981 | controller | - ' fi' 2026-03-09 19:10:03.734985 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.734990 | controller | - ' ' 2026-03-09 19:10:03.735001 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.735006 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.735012 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735021 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.735029 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735035 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.735041 | controller | -- \' 2026-03-09 19:10:03.735045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735050 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.735054 | controller | - ' logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.735058 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.735064 | controller | - ' logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.735069 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.735073 | controller | found' 2026-03-09 19:10:03.735077 | controller | - ' logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.735082 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.735086 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.735090 | controller | - ' ' 2026-03-09 19:10:03.735095 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.735099 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.735103 | controller | - ' ' 2026-03-09 19:10:03.735107 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.735112 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.735116 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.735120 | controller | - ' exit 1' 2026-03-09 19:10:03.735125 | controller | - ' fi' 2026-03-09 19:10:03.735129 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.735133 | controller | - ' ' 2026-03-09 19:10:03.735137 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.735142 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.735165 | controller | | base64 -d)' 2026-03-09 19:10:03.735170 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.735176 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.735180 | controller | - ' exit 1' 2026-03-09 19:10:03.735185 | controller | - ' fi' 2026-03-09 19:10:03.735189 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.735193 | controller | - ' ' 2026-03-09 19:10:03.735197 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.735202 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.735206 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.735210 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.735215 | controller | --timeout=300s' 2026-03-09 19:10:03.735219 | controller | - ' ' 2026-03-09 19:10:03.735223 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.735227 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.735232 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.735236 | controller | - ' ' 2026-03-09 19:10:03.735242 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735246 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735251 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735255 | controller | - ' ' 2026-03-09 19:10:03.735259 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.735264 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.735268 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.735272 | controller | available"' 2026-03-09 19:10:03.735277 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.735281 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.735285 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.735290 | controller | - ' exit 1' 2026-03-09 19:10:03.735294 | controller | - ' fi' 2026-03-09 19:10:03.735298 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.735303 | controller | - ' ' 2026-03-09 19:10:03.735307 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.735319 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.735324 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735328 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.735332 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735336 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.735341 | controller | -- \' 2026-03-09 19:10:03.735345 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735349 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.735354 | controller | - ' logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.735358 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.735362 | controller | - ' logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.735367 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.735371 | controller | found' 2026-03-09 19:10:03.735377 | controller | - ' logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.735381 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.735386 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.735390 | controller | - ' ' 2026-03-09 19:10:03.735394 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.735399 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.735403 | controller | - ' ' 2026-03-09 19:10:03.735407 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.735412 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.735416 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.735420 | controller | - ' exit 1' 2026-03-09 19:10:03.735424 | controller | - ' fi' 2026-03-09 19:10:03.735429 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.735433 | controller | - ' ' 2026-03-09 19:10:03.735439 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.735443 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.735448 | controller | | base64 -d)' 2026-03-09 19:10:03.735452 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.735456 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.735460 | controller | - ' exit 1' 2026-03-09 19:10:03.735465 | controller | - ' fi' 2026-03-09 19:10:03.735469 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.735473 | controller | - ' ' 2026-03-09 19:10:03.735478 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.735482 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.735486 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.735491 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.735495 | controller | --timeout=300s' 2026-03-09 19:10:03.735499 | controller | - ' ' 2026-03-09 19:10:03.735503 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.735508 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.735512 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.735516 | controller | - ' ' 2026-03-09 19:10:03.735521 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735525 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735529 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735533 | controller | - ' ' 2026-03-09 19:10:03.735538 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.735542 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.735546 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.735551 | controller | available"' 2026-03-09 19:10:03.735555 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.735559 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.735563 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.735568 | controller | - ' exit 1' 2026-03-09 19:10:03.735572 | controller | - ' fi' 2026-03-09 19:10:03.735576 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.735581 | controller | - ' ' 2026-03-09 19:10:03.735585 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.735597 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.735601 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735606 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.735610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735614 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.735619 | controller | -- \' 2026-03-09 19:10:03.735623 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735629 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.735634 | controller | - ' logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.735638 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.735642 | controller | - ' logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.735648 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.735653 | controller | found' 2026-03-09 19:10:03.735657 | controller | - ' logger.go:42: 18:54:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.735661 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.735666 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.735670 | controller | - ' ' 2026-03-09 19:10:03.735674 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.735679 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.735683 | controller | - ' ' 2026-03-09 19:10:03.735687 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.735708 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.735713 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.735717 | controller | - ' exit 1' 2026-03-09 19:10:03.735722 | controller | - ' fi' 2026-03-09 19:10:03.735726 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.735730 | controller | - ' ' 2026-03-09 19:10:03.735735 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.735739 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.735743 | controller | | base64 -d)' 2026-03-09 19:10:03.735748 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.735752 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.735756 | controller | - ' exit 1' 2026-03-09 19:10:03.735760 | controller | - ' fi' 2026-03-09 19:10:03.735765 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.735769 | controller | - ' ' 2026-03-09 19:10:03.735773 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.735778 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.735782 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.735786 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.735791 | controller | --timeout=300s' 2026-03-09 19:10:03.735797 | controller | - ' ' 2026-03-09 19:10:03.735801 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.735806 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.735810 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.735814 | controller | - ' ' 2026-03-09 19:10:03.735819 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735823 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735827 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.735832 | controller | - ' ' 2026-03-09 19:10:03.735836 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.735840 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.735848 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.735853 | controller | available"' 2026-03-09 19:10:03.735857 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.735861 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.735865 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.735870 | controller | - ' exit 1' 2026-03-09 19:10:03.735874 | controller | - ' fi' 2026-03-09 19:10:03.735878 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.735883 | controller | - ' ' 2026-03-09 19:10:03.735887 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.735891 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.735903 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735908 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.735912 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735916 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.735920 | controller | -- \' 2026-03-09 19:10:03.735925 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.735929 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.735934 | controller | - ' logger.go:42: 18:54:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.735938 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.735945 | controller | - ' logger.go:42: 18:55:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.735949 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.735953 | controller | found' 2026-03-09 19:10:03.735958 | controller | - ' logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.735962 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.735966 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.735971 | controller | - ' ' 2026-03-09 19:10:03.735975 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.735979 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.735984 | controller | - ' ' 2026-03-09 19:10:03.735988 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.735992 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.735997 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.736001 | controller | - ' exit 1' 2026-03-09 19:10:03.736006 | controller | - ' fi' 2026-03-09 19:10:03.736012 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.736018 | controller | - ' ' 2026-03-09 19:10:03.736027 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.736033 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.736039 | controller | | base64 -d)' 2026-03-09 19:10:03.736043 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.736047 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.736051 | controller | - ' exit 1' 2026-03-09 19:10:03.736056 | controller | - ' fi' 2026-03-09 19:10:03.736060 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.736066 | controller | - ' ' 2026-03-09 19:10:03.736071 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.736075 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.736079 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.736084 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.736088 | controller | --timeout=300s' 2026-03-09 19:10:03.736092 | controller | - ' ' 2026-03-09 19:10:03.736097 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.736101 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.736105 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.736109 | controller | - ' ' 2026-03-09 19:10:03.736114 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736124 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736129 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736133 | controller | - ' ' 2026-03-09 19:10:03.736138 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.736142 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.736162 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.736166 | controller | available"' 2026-03-09 19:10:03.736171 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.736175 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.736179 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.736183 | controller | - ' exit 1' 2026-03-09 19:10:03.736188 | controller | - ' fi' 2026-03-09 19:10:03.736192 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.736196 | controller | - ' ' 2026-03-09 19:10:03.736201 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.736205 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.736219 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736223 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.736228 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736232 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.736236 | controller | -- \' 2026-03-09 19:10:03.736241 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736245 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.736249 | controller | - ' logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.736254 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.736258 | controller | - ' logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.736262 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.736269 | controller | found' 2026-03-09 19:10:03.736275 | controller | - ' logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.736280 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.736284 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.736288 | controller | - ' ' 2026-03-09 19:10:03.736293 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.736297 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.736301 | controller | - ' ' 2026-03-09 19:10:03.736306 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.736310 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.736314 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.736319 | controller | - ' exit 1' 2026-03-09 19:10:03.736323 | controller | - ' fi' 2026-03-09 19:10:03.736327 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.736332 | controller | - ' ' 2026-03-09 19:10:03.736336 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.736340 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.736345 | controller | | base64 -d)' 2026-03-09 19:10:03.736349 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.736353 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.736357 | controller | - ' exit 1' 2026-03-09 19:10:03.736362 | controller | - ' fi' 2026-03-09 19:10:03.736366 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.736370 | controller | - ' ' 2026-03-09 19:10:03.736375 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.736379 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.736383 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.736387 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.736392 | controller | --timeout=300s' 2026-03-09 19:10:03.736396 | controller | - ' ' 2026-03-09 19:10:03.736400 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.736405 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.736409 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.736413 | controller | - ' ' 2026-03-09 19:10:03.736417 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736422 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736426 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736430 | controller | - ' ' 2026-03-09 19:10:03.736434 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.736439 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.736443 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.736447 | controller | available"' 2026-03-09 19:10:03.736452 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.736456 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.736460 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.736465 | controller | - ' exit 1' 2026-03-09 19:10:03.736471 | controller | - ' fi' 2026-03-09 19:10:03.736475 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.736480 | controller | - ' ' 2026-03-09 19:10:03.736484 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.736488 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.736493 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736504 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.736509 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736513 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.736517 | controller | -- \' 2026-03-09 19:10:03.736522 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736526 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.736532 | controller | - ' logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.736537 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.736541 | controller | - ' logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.736545 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.736550 | controller | found' 2026-03-09 19:10:03.736554 | controller | - ' logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.736558 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.736562 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.736567 | controller | - ' ' 2026-03-09 19:10:03.736571 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.736575 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.736580 | controller | - ' ' 2026-03-09 19:10:03.736584 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.736588 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.736593 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.736597 | controller | - ' exit 1' 2026-03-09 19:10:03.736601 | controller | - ' fi' 2026-03-09 19:10:03.736605 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.736610 | controller | - ' ' 2026-03-09 19:10:03.736614 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.736618 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.736623 | controller | | base64 -d)' 2026-03-09 19:10:03.736627 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.736631 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.736636 | controller | - ' exit 1' 2026-03-09 19:10:03.736640 | controller | - ' fi' 2026-03-09 19:10:03.736644 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.736649 | controller | - ' ' 2026-03-09 19:10:03.736653 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.736657 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.736662 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.736666 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.736670 | controller | --timeout=300s' 2026-03-09 19:10:03.736678 | controller | - ' ' 2026-03-09 19:10:03.736682 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.736687 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.736691 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.736695 | controller | - ' ' 2026-03-09 19:10:03.736700 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736704 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736708 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.736713 | controller | - ' ' 2026-03-09 19:10:03.736717 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.736721 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.736726 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.736730 | controller | available"' 2026-03-09 19:10:03.736734 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.736739 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.736743 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.736747 | controller | - ' exit 1' 2026-03-09 19:10:03.736751 | controller | - ' fi' 2026-03-09 19:10:03.736756 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.736760 | controller | - ' ' 2026-03-09 19:10:03.736764 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.736769 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.736773 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736784 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.736789 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736793 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.736797 | controller | -- \' 2026-03-09 19:10:03.736802 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.736806 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.736811 | controller | - ' logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.736815 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.736819 | controller | - ' logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.736823 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.736828 | controller | found' 2026-03-09 19:10:03.736834 | controller | - ' logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.736840 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.736846 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.736851 | controller | - ' ' 2026-03-09 19:10:03.736856 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.736862 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.736867 | controller | - ' ' 2026-03-09 19:10:03.736876 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.736882 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.736887 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.736893 | controller | - ' exit 1' 2026-03-09 19:10:03.736901 | controller | - ' fi' 2026-03-09 19:10:03.736907 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.736913 | controller | - ' ' 2026-03-09 19:10:03.736920 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.736926 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.736932 | controller | | base64 -d)' 2026-03-09 19:10:03.736937 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.736943 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.736948 | controller | - ' exit 1' 2026-03-09 19:10:03.736954 | controller | - ' fi' 2026-03-09 19:10:03.736960 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.736966 | controller | - ' ' 2026-03-09 19:10:03.736971 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.736977 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.736983 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.736988 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.736994 | controller | --timeout=300s' 2026-03-09 19:10:03.737000 | controller | - ' ' 2026-03-09 19:10:03.737005 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.737011 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.737017 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.737023 | controller | - ' ' 2026-03-09 19:10:03.737029 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737035 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737040 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737044 | controller | - ' ' 2026-03-09 19:10:03.737048 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.737052 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.737057 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.737061 | controller | available"' 2026-03-09 19:10:03.737065 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.737069 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.737074 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.737078 | controller | - ' exit 1' 2026-03-09 19:10:03.737082 | controller | - ' fi' 2026-03-09 19:10:03.737086 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.737090 | controller | - ' ' 2026-03-09 19:10:03.737095 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.737099 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.737103 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737110 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.737125 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737130 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.737138 | controller | -- \' 2026-03-09 19:10:03.737142 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737163 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.737168 | controller | - ' logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.737172 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.737187 | controller | - ' logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.737191 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.737195 | controller | found' 2026-03-09 19:10:03.737199 | controller | - ' logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.737204 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.737208 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.737212 | controller | - ' ' 2026-03-09 19:10:03.737216 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.737221 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.737225 | controller | - ' ' 2026-03-09 19:10:03.737229 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.737233 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.737238 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.737242 | controller | - ' exit 1' 2026-03-09 19:10:03.737246 | controller | - ' fi' 2026-03-09 19:10:03.737250 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.737255 | controller | - ' ' 2026-03-09 19:10:03.737259 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.737263 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.737267 | controller | | base64 -d)' 2026-03-09 19:10:03.737272 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.737276 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.737280 | controller | - ' exit 1' 2026-03-09 19:10:03.737284 | controller | - ' fi' 2026-03-09 19:10:03.737289 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.737293 | controller | - ' ' 2026-03-09 19:10:03.737297 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.737301 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.737306 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.737310 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.737314 | controller | --timeout=300s' 2026-03-09 19:10:03.737318 | controller | - ' ' 2026-03-09 19:10:03.737323 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.737327 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.737331 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.737335 | controller | - ' ' 2026-03-09 19:10:03.737339 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737346 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737351 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737355 | controller | - ' ' 2026-03-09 19:10:03.737359 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.737364 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.737368 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.737372 | controller | available"' 2026-03-09 19:10:03.737376 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.737383 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.737387 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.737391 | controller | - ' exit 1' 2026-03-09 19:10:03.737396 | controller | - ' fi' 2026-03-09 19:10:03.737400 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.737404 | controller | - ' ' 2026-03-09 19:10:03.737408 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.737413 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.737417 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737421 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.737434 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737439 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.737443 | controller | -- \' 2026-03-09 19:10:03.737447 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737451 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.737456 | controller | - ' logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.737460 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.737464 | controller | - ' logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.737469 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.737473 | controller | found' 2026-03-09 19:10:03.737477 | controller | - ' logger.go:42: 18:55:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.737481 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.737486 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.737490 | controller | - ' ' 2026-03-09 19:10:03.737494 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.737500 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.737505 | controller | - ' ' 2026-03-09 19:10:03.737509 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.737513 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.737518 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.737522 | controller | - ' exit 1' 2026-03-09 19:10:03.737526 | controller | - ' fi' 2026-03-09 19:10:03.737530 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.737535 | controller | - ' ' 2026-03-09 19:10:03.737539 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.737545 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.737549 | controller | | base64 -d)' 2026-03-09 19:10:03.737554 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.737558 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.737562 | controller | - ' exit 1' 2026-03-09 19:10:03.737566 | controller | - ' fi' 2026-03-09 19:10:03.737571 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.737575 | controller | - ' ' 2026-03-09 19:10:03.737579 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.737583 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.737588 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.737592 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.737596 | controller | --timeout=300s' 2026-03-09 19:10:03.737600 | controller | - ' ' 2026-03-09 19:10:03.737605 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.737609 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.737613 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.737618 | controller | - ' ' 2026-03-09 19:10:03.737622 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737626 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737630 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737635 | controller | - ' ' 2026-03-09 19:10:03.737639 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.737643 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.737648 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.737652 | controller | available"' 2026-03-09 19:10:03.737656 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.737660 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.737665 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.737669 | controller | - ' exit 1' 2026-03-09 19:10:03.737673 | controller | - ' fi' 2026-03-09 19:10:03.737677 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.737682 | controller | - ' ' 2026-03-09 19:10:03.737686 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.737690 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.737694 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737699 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.737703 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737715 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.737719 | controller | -- \' 2026-03-09 19:10:03.737723 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737728 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.737734 | controller | - ' logger.go:42: 18:55:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.737738 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.737743 | controller | - ' logger.go:42: 18:55:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.737747 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.737751 | controller | found' 2026-03-09 19:10:03.737755 | controller | - ' logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.737760 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.737764 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.737768 | controller | - ' ' 2026-03-09 19:10:03.737773 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.737777 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.737781 | controller | - ' ' 2026-03-09 19:10:03.737785 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.737790 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.737794 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.737798 | controller | - ' exit 1' 2026-03-09 19:10:03.737802 | controller | - ' fi' 2026-03-09 19:10:03.737807 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.737811 | controller | - ' ' 2026-03-09 19:10:03.737815 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.737820 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.737824 | controller | | base64 -d)' 2026-03-09 19:10:03.737828 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.737832 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.737837 | controller | - ' exit 1' 2026-03-09 19:10:03.737841 | controller | - ' fi' 2026-03-09 19:10:03.737845 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.737850 | controller | - ' ' 2026-03-09 19:10:03.737854 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.737858 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.737862 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.737867 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.737871 | controller | --timeout=300s' 2026-03-09 19:10:03.737875 | controller | - ' ' 2026-03-09 19:10:03.737880 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.737884 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.737888 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.737892 | controller | - ' ' 2026-03-09 19:10:03.737899 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737903 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737907 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.737912 | controller | - ' ' 2026-03-09 19:10:03.737916 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.737920 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.737926 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.737930 | controller | available"' 2026-03-09 19:10:03.737934 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.737939 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.737943 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.737947 | controller | - ' exit 1' 2026-03-09 19:10:03.737951 | controller | - ' fi' 2026-03-09 19:10:03.737956 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.737960 | controller | - ' ' 2026-03-09 19:10:03.737966 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.737970 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.737975 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737979 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.737983 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.737995 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.737999 | controller | -- \' 2026-03-09 19:10:03.738004 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738009 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.738015 | controller | - ' logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.738021 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.738027 | controller | - ' logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.738034 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.738040 | controller | found' 2026-03-09 19:10:03.738044 | controller | - ' logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.738049 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.738053 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.738057 | controller | - ' ' 2026-03-09 19:10:03.738062 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.738066 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.738071 | controller | - ' ' 2026-03-09 19:10:03.738080 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.738085 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.738089 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.738093 | controller | - ' exit 1' 2026-03-09 19:10:03.738097 | controller | - ' fi' 2026-03-09 19:10:03.738102 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.738106 | controller | - ' ' 2026-03-09 19:10:03.738110 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.738115 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.738119 | controller | | base64 -d)' 2026-03-09 19:10:03.738123 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.738127 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.738132 | controller | - ' exit 1' 2026-03-09 19:10:03.738136 | controller | - ' fi' 2026-03-09 19:10:03.738140 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.738164 | controller | - ' ' 2026-03-09 19:10:03.738171 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.738176 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.738180 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.738184 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.738188 | controller | --timeout=300s' 2026-03-09 19:10:03.738193 | controller | - ' ' 2026-03-09 19:10:03.738197 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.738201 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.738205 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.738210 | controller | - ' ' 2026-03-09 19:10:03.738214 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738218 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738225 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738229 | controller | - ' ' 2026-03-09 19:10:03.738233 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.738237 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.738242 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.738246 | controller | available"' 2026-03-09 19:10:03.738250 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.738254 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.738259 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.738263 | controller | - ' exit 1' 2026-03-09 19:10:03.738267 | controller | - ' fi' 2026-03-09 19:10:03.738272 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.738276 | controller | - ' ' 2026-03-09 19:10:03.738280 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.738286 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.738291 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738296 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.738300 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738304 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.738308 | controller | -- \' 2026-03-09 19:10:03.738323 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738328 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.738332 | controller | - ' logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.738336 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.738341 | controller | - ' logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.738345 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.738349 | controller | found' 2026-03-09 19:10:03.738354 | controller | - ' logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.738360 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.738364 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.738369 | controller | - ' ' 2026-03-09 19:10:03.738373 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.738377 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.738382 | controller | - ' ' 2026-03-09 19:10:03.738386 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.738390 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.738394 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.738399 | controller | - ' exit 1' 2026-03-09 19:10:03.738403 | controller | - ' fi' 2026-03-09 19:10:03.738407 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.738411 | controller | - ' ' 2026-03-09 19:10:03.738416 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.738420 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.738424 | controller | | base64 -d)' 2026-03-09 19:10:03.738429 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.738433 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.738437 | controller | - ' exit 1' 2026-03-09 19:10:03.738441 | controller | - ' fi' 2026-03-09 19:10:03.738446 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.738450 | controller | - ' ' 2026-03-09 19:10:03.738454 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.738458 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.738463 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.738467 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.738471 | controller | --timeout=300s' 2026-03-09 19:10:03.738475 | controller | - ' ' 2026-03-09 19:10:03.738480 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.738484 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.738488 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.738492 | controller | - ' ' 2026-03-09 19:10:03.738497 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738501 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738505 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738510 | controller | - ' ' 2026-03-09 19:10:03.738514 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.738518 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.738522 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.738527 | controller | available"' 2026-03-09 19:10:03.738531 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.738535 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.738539 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.738543 | controller | - ' exit 1' 2026-03-09 19:10:03.738548 | controller | - ' fi' 2026-03-09 19:10:03.738552 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.738558 | controller | - ' ' 2026-03-09 19:10:03.738562 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.738567 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.738571 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738575 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.738579 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738584 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.738588 | controller | -- \' 2026-03-09 19:10:03.738599 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738603 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.738608 | controller | - ' logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.738612 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.738616 | controller | - ' logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.738621 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.738625 | controller | found' 2026-03-09 19:10:03.738629 | controller | - ' logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.738633 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.738638 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.738642 | controller | - ' ' 2026-03-09 19:10:03.738646 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.738651 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.738655 | controller | - ' ' 2026-03-09 19:10:03.738659 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.738663 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.738668 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.738673 | controller | - ' exit 1' 2026-03-09 19:10:03.738679 | controller | - ' fi' 2026-03-09 19:10:03.738683 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.738690 | controller | - ' ' 2026-03-09 19:10:03.738694 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.738698 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.738702 | controller | | base64 -d)' 2026-03-09 19:10:03.738707 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.738714 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.738718 | controller | - ' exit 1' 2026-03-09 19:10:03.738723 | controller | - ' fi' 2026-03-09 19:10:03.738727 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.738731 | controller | - ' ' 2026-03-09 19:10:03.738735 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.738740 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.738744 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.738748 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.738752 | controller | --timeout=300s' 2026-03-09 19:10:03.738757 | controller | - ' ' 2026-03-09 19:10:03.738761 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.738767 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.738772 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.738776 | controller | - ' ' 2026-03-09 19:10:03.738780 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738786 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738791 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.738795 | controller | - ' ' 2026-03-09 19:10:03.738800 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.738804 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.738808 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.738813 | controller | available"' 2026-03-09 19:10:03.738817 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.738821 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.738825 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.738830 | controller | - ' exit 1' 2026-03-09 19:10:03.738834 | controller | - ' fi' 2026-03-09 19:10:03.738838 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.738842 | controller | - ' ' 2026-03-09 19:10:03.738847 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.738851 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.738855 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738860 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.738864 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738868 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.738872 | controller | -- \' 2026-03-09 19:10:03.738877 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.738888 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.738893 | controller | - ' logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.738897 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.738901 | controller | - ' logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.738906 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.738910 | controller | found' 2026-03-09 19:10:03.738914 | controller | - ' logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.738919 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.738923 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.738927 | controller | - ' ' 2026-03-09 19:10:03.738932 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.738936 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.738940 | controller | - ' ' 2026-03-09 19:10:03.738944 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.738951 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.738955 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.738960 | controller | - ' exit 1' 2026-03-09 19:10:03.738964 | controller | - ' fi' 2026-03-09 19:10:03.738968 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.738973 | controller | - ' ' 2026-03-09 19:10:03.738977 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.738981 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.738985 | controller | | base64 -d)' 2026-03-09 19:10:03.738990 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.738994 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.738998 | controller | - ' exit 1' 2026-03-09 19:10:03.739003 | controller | - ' fi' 2026-03-09 19:10:03.739008 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.739014 | controller | - ' ' 2026-03-09 19:10:03.739020 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.739026 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.739032 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.739038 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.739043 | controller | --timeout=300s' 2026-03-09 19:10:03.739048 | controller | - ' ' 2026-03-09 19:10:03.739052 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.739056 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.739061 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.739065 | controller | - ' ' 2026-03-09 19:10:03.739072 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739076 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739080 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739085 | controller | - ' ' 2026-03-09 19:10:03.739089 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.739093 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.739097 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.739102 | controller | available"' 2026-03-09 19:10:03.739106 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.739110 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.739114 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.739119 | controller | - ' exit 1' 2026-03-09 19:10:03.739123 | controller | - ' fi' 2026-03-09 19:10:03.739127 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.739132 | controller | - ' ' 2026-03-09 19:10:03.739136 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.739140 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.739158 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739165 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.739171 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739176 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.739180 | controller | -- \' 2026-03-09 19:10:03.739184 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739197 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.739204 | controller | - ' logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.739208 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.739212 | controller | - ' logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.739217 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.739221 | controller | found' 2026-03-09 19:10:03.739225 | controller | - ' logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.739230 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.739234 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.739238 | controller | - ' ' 2026-03-09 19:10:03.739242 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.739247 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.739251 | controller | - ' ' 2026-03-09 19:10:03.739255 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.739259 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.739264 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.739268 | controller | - ' exit 1' 2026-03-09 19:10:03.739272 | controller | - ' fi' 2026-03-09 19:10:03.739277 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.739281 | controller | - ' ' 2026-03-09 19:10:03.739285 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.739289 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.739294 | controller | | base64 -d)' 2026-03-09 19:10:03.739298 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.739302 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.739308 | controller | - ' exit 1' 2026-03-09 19:10:03.739313 | controller | - ' fi' 2026-03-09 19:10:03.739317 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.739321 | controller | - ' ' 2026-03-09 19:10:03.739325 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.739330 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.739334 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.739340 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.739344 | controller | --timeout=300s' 2026-03-09 19:10:03.739348 | controller | - ' ' 2026-03-09 19:10:03.739353 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.739357 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.739361 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.739365 | controller | - ' ' 2026-03-09 19:10:03.739370 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739376 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739380 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739384 | controller | - ' ' 2026-03-09 19:10:03.739388 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.739393 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.739397 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.739401 | controller | available"' 2026-03-09 19:10:03.739406 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.739410 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.739414 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.739418 | controller | - ' exit 1' 2026-03-09 19:10:03.739423 | controller | - ' fi' 2026-03-09 19:10:03.739427 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.739431 | controller | - ' ' 2026-03-09 19:10:03.739435 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.739440 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.739444 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.739452 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739457 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.739461 | controller | -- \' 2026-03-09 19:10:03.739465 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739470 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.739481 | controller | - ' logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.739486 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.739490 | controller | - ' logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.739495 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.739499 | controller | found' 2026-03-09 19:10:03.739503 | controller | - ' logger.go:42: 18:55:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.739508 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.739512 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.739519 | controller | - ' ' 2026-03-09 19:10:03.739523 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.739527 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.739532 | controller | - ' ' 2026-03-09 19:10:03.739536 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.739540 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.739545 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.739549 | controller | - ' exit 1' 2026-03-09 19:10:03.739554 | controller | - ' fi' 2026-03-09 19:10:03.739558 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.739563 | controller | - ' ' 2026-03-09 19:10:03.739567 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.739571 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.739577 | controller | | base64 -d)' 2026-03-09 19:10:03.739582 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.739586 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.739591 | controller | - ' exit 1' 2026-03-09 19:10:03.739595 | controller | - ' fi' 2026-03-09 19:10:03.739599 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.739604 | controller | - ' ' 2026-03-09 19:10:03.739608 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.739613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.739617 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.739621 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.739626 | controller | --timeout=300s' 2026-03-09 19:10:03.739630 | controller | - ' ' 2026-03-09 19:10:03.739634 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.739639 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.739643 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.739647 | controller | - ' ' 2026-03-09 19:10:03.739652 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739656 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739661 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739665 | controller | - ' ' 2026-03-09 19:10:03.739669 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.739674 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.739678 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.739682 | controller | available"' 2026-03-09 19:10:03.739703 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.739710 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.739715 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.739719 | controller | - ' exit 1' 2026-03-09 19:10:03.739723 | controller | - ' fi' 2026-03-09 19:10:03.739728 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.739732 | controller | - ' ' 2026-03-09 19:10:03.739737 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.739741 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.739745 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739750 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.739754 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739758 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.739763 | controller | -- \' 2026-03-09 19:10:03.739767 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.739771 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.739785 | controller | - ' logger.go:42: 18:55:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.739790 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.739794 | controller | - ' logger.go:42: 18:55:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.739799 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.739803 | controller | found' 2026-03-09 19:10:03.739807 | controller | - ' logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.739812 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.739816 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.739820 | controller | - ' ' 2026-03-09 19:10:03.739825 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.739829 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.739835 | controller | - ' ' 2026-03-09 19:10:03.739841 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.739847 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.739853 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.739859 | controller | - ' exit 1' 2026-03-09 19:10:03.739864 | controller | - ' fi' 2026-03-09 19:10:03.739868 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.739872 | controller | - ' ' 2026-03-09 19:10:03.739876 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.739881 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.739885 | controller | | base64 -d)' 2026-03-09 19:10:03.739889 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.739894 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.739898 | controller | - ' exit 1' 2026-03-09 19:10:03.739902 | controller | - ' fi' 2026-03-09 19:10:03.739906 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.739911 | controller | - ' ' 2026-03-09 19:10:03.739915 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.739919 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.739923 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.739930 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.739934 | controller | --timeout=300s' 2026-03-09 19:10:03.739939 | controller | - ' ' 2026-03-09 19:10:03.739943 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.739947 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.739952 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.739956 | controller | - ' ' 2026-03-09 19:10:03.739960 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739964 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739969 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.739973 | controller | - ' ' 2026-03-09 19:10:03.739977 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.739982 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.739986 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.739992 | controller | available"' 2026-03-09 19:10:03.739996 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.740004 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.740009 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.740015 | controller | - ' exit 1' 2026-03-09 19:10:03.740021 | controller | - ' fi' 2026-03-09 19:10:03.740027 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.740033 | controller | - ' ' 2026-03-09 19:10:03.740039 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.740046 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.740050 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740054 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.740059 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740063 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.740067 | controller | -- \' 2026-03-09 19:10:03.740071 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740076 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.740080 | controller | - ' logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.740093 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.740098 | controller | - ' logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.740102 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.740106 | controller | found' 2026-03-09 19:10:03.740111 | controller | - ' logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.740115 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.740119 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.740123 | controller | - ' ' 2026-03-09 19:10:03.740128 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.740132 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.740136 | controller | - ' ' 2026-03-09 19:10:03.740141 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.740157 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.740164 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.740168 | controller | - ' exit 1' 2026-03-09 19:10:03.740173 | controller | - ' fi' 2026-03-09 19:10:03.740177 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.740181 | controller | - ' ' 2026-03-09 19:10:03.740186 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.740190 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.740194 | controller | | base64 -d)' 2026-03-09 19:10:03.740199 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.740203 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.740207 | controller | - ' exit 1' 2026-03-09 19:10:03.740214 | controller | - ' fi' 2026-03-09 19:10:03.740218 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.740223 | controller | - ' ' 2026-03-09 19:10:03.740227 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.740234 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.740238 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.740242 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.740246 | controller | --timeout=300s' 2026-03-09 19:10:03.740251 | controller | - ' ' 2026-03-09 19:10:03.740255 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.740259 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.740264 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.740270 | controller | - ' ' 2026-03-09 19:10:03.740274 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740278 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740283 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740287 | controller | - ' ' 2026-03-09 19:10:03.740291 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.740297 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.740302 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.740306 | controller | available"' 2026-03-09 19:10:03.740310 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.740315 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.740319 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.740323 | controller | - ' exit 1' 2026-03-09 19:10:03.740327 | controller | - ' fi' 2026-03-09 19:10:03.740332 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.740336 | controller | - ' ' 2026-03-09 19:10:03.740342 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.740348 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.740353 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740359 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.740364 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740370 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.740376 | controller | -- \' 2026-03-09 19:10:03.740381 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740387 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.740393 | controller | - ' logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.740398 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.740415 | controller | - ' logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.740422 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.740429 | controller | found' 2026-03-09 19:10:03.740435 | controller | - ' logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.740443 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.740449 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.740455 | controller | - ' ' 2026-03-09 19:10:03.740461 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.740466 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.740472 | controller | - ' ' 2026-03-09 19:10:03.740477 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.740483 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.740489 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.740495 | controller | - ' exit 1' 2026-03-09 19:10:03.740500 | controller | - ' fi' 2026-03-09 19:10:03.740506 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.740511 | controller | - ' ' 2026-03-09 19:10:03.740517 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.740523 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.740529 | controller | | base64 -d)' 2026-03-09 19:10:03.740534 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.740538 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.740542 | controller | - ' exit 1' 2026-03-09 19:10:03.740546 | controller | - ' fi' 2026-03-09 19:10:03.740551 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.740555 | controller | - ' ' 2026-03-09 19:10:03.740559 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.740564 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.740568 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.740572 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.740576 | controller | --timeout=300s' 2026-03-09 19:10:03.740581 | controller | - ' ' 2026-03-09 19:10:03.740585 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.740589 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.740593 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.740598 | controller | - ' ' 2026-03-09 19:10:03.740602 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740606 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740610 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740615 | controller | - ' ' 2026-03-09 19:10:03.740622 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.740627 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.740631 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.740635 | controller | available"' 2026-03-09 19:10:03.740640 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.740644 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.740648 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.740653 | controller | - ' exit 1' 2026-03-09 19:10:03.740657 | controller | - ' fi' 2026-03-09 19:10:03.740661 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.740668 | controller | - ' ' 2026-03-09 19:10:03.740672 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.740676 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.740681 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740685 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.740689 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740693 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.740698 | controller | -- \' 2026-03-09 19:10:03.740702 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740706 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.740712 | controller | - ' logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.740716 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.740721 | controller | - ' logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.740735 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.740739 | controller | found' 2026-03-09 19:10:03.740743 | controller | - ' logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.740748 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.740752 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.740757 | controller | - ' ' 2026-03-09 19:10:03.740761 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.740765 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.740769 | controller | - ' ' 2026-03-09 19:10:03.740774 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.740778 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.740782 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.740787 | controller | - ' exit 1' 2026-03-09 19:10:03.740791 | controller | - ' fi' 2026-03-09 19:10:03.740795 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.740799 | controller | - ' ' 2026-03-09 19:10:03.740804 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.740808 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.740812 | controller | | base64 -d)' 2026-03-09 19:10:03.740817 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.740821 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.740825 | controller | - ' exit 1' 2026-03-09 19:10:03.740829 | controller | - ' fi' 2026-03-09 19:10:03.740834 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.740838 | controller | - ' ' 2026-03-09 19:10:03.740842 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.740846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.740851 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.740855 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.740859 | controller | --timeout=300s' 2026-03-09 19:10:03.740865 | controller | - ' ' 2026-03-09 19:10:03.740870 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.740876 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.740880 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.740885 | controller | - ' ' 2026-03-09 19:10:03.740889 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740893 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740898 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.740902 | controller | - ' ' 2026-03-09 19:10:03.740906 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.740910 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.740915 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.740919 | controller | available"' 2026-03-09 19:10:03.740923 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.740927 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.740932 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.740936 | controller | - ' exit 1' 2026-03-09 19:10:03.740940 | controller | - ' fi' 2026-03-09 19:10:03.740945 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.740949 | controller | - ' ' 2026-03-09 19:10:03.740953 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.740957 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.740962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740966 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.740970 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740974 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.740979 | controller | -- \' 2026-03-09 19:10:03.740983 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.740987 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.740991 | controller | - ' logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.740996 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.741000 | controller | - ' logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.741013 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.741020 | controller | found' 2026-03-09 19:10:03.741026 | controller | - ' logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.741032 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.741038 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.741043 | controller | - ' ' 2026-03-09 19:10:03.741047 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.741051 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.741055 | controller | - ' ' 2026-03-09 19:10:03.741060 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.741064 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.741071 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.741075 | controller | - ' exit 1' 2026-03-09 19:10:03.741079 | controller | - ' fi' 2026-03-09 19:10:03.741084 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.741088 | controller | - ' ' 2026-03-09 19:10:03.741092 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.741096 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.741101 | controller | | base64 -d)' 2026-03-09 19:10:03.741105 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.741109 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.741113 | controller | - ' exit 1' 2026-03-09 19:10:03.741118 | controller | - ' fi' 2026-03-09 19:10:03.741122 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.741126 | controller | - ' ' 2026-03-09 19:10:03.741131 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.741135 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.741139 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.741143 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.741167 | controller | --timeout=300s' 2026-03-09 19:10:03.741172 | controller | - ' ' 2026-03-09 19:10:03.741176 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.741180 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.741185 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.741189 | controller | - ' ' 2026-03-09 19:10:03.741193 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741197 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741202 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741206 | controller | - ' ' 2026-03-09 19:10:03.741210 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.741214 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.741221 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.741225 | controller | available"' 2026-03-09 19:10:03.741230 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.741234 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.741238 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.741242 | controller | - ' exit 1' 2026-03-09 19:10:03.741247 | controller | - ' fi' 2026-03-09 19:10:03.741251 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.741255 | controller | - ' ' 2026-03-09 19:10:03.741259 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.741264 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.741268 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741272 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.741276 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741284 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.741288 | controller | -- \' 2026-03-09 19:10:03.741292 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741297 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.741301 | controller | - ' logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.741305 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.741310 | controller | - ' logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.741314 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.741318 | controller | found' 2026-03-09 19:10:03.741331 | controller | - ' logger.go:42: 18:55:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.741336 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.741340 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.741344 | controller | - ' ' 2026-03-09 19:10:03.741349 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.741353 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.741357 | controller | - ' ' 2026-03-09 19:10:03.741364 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.741368 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.741372 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.741377 | controller | - ' exit 1' 2026-03-09 19:10:03.741381 | controller | - ' fi' 2026-03-09 19:10:03.741385 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.741390 | controller | - ' ' 2026-03-09 19:10:03.741397 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.741401 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.741406 | controller | | base64 -d)' 2026-03-09 19:10:03.741410 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.741414 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.741419 | controller | - ' exit 1' 2026-03-09 19:10:03.741423 | controller | - ' fi' 2026-03-09 19:10:03.741427 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.741432 | controller | - ' ' 2026-03-09 19:10:03.741436 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.741440 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.741444 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.741449 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.741453 | controller | --timeout=300s' 2026-03-09 19:10:03.741457 | controller | - ' ' 2026-03-09 19:10:03.741462 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.741466 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.741472 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.741476 | controller | - ' ' 2026-03-09 19:10:03.741480 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741485 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741491 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741496 | controller | - ' ' 2026-03-09 19:10:03.741500 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.741504 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.741509 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.741529 | controller | available"' 2026-03-09 19:10:03.741534 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.741538 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.741542 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.741546 | controller | - ' exit 1' 2026-03-09 19:10:03.741551 | controller | - ' fi' 2026-03-09 19:10:03.741555 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.741559 | controller | - ' ' 2026-03-09 19:10:03.741564 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.741568 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.741572 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741576 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.741581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741585 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.741589 | controller | -- \' 2026-03-09 19:10:03.741593 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741598 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.741602 | controller | - ' logger.go:42: 18:55:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.741606 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.741611 | controller | - ' logger.go:42: 18:55:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.741615 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.741619 | controller | found' 2026-03-09 19:10:03.741631 | controller | - ' logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.741635 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.741640 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.741644 | controller | - ' ' 2026-03-09 19:10:03.741648 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.741653 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.741657 | controller | - ' ' 2026-03-09 19:10:03.741661 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.741666 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.741670 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.741674 | controller | - ' exit 1' 2026-03-09 19:10:03.741678 | controller | - ' fi' 2026-03-09 19:10:03.741683 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.741687 | controller | - ' ' 2026-03-09 19:10:03.741691 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.741696 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.741702 | controller | | base64 -d)' 2026-03-09 19:10:03.741707 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.741711 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.741715 | controller | - ' exit 1' 2026-03-09 19:10:03.741720 | controller | - ' fi' 2026-03-09 19:10:03.741724 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.741728 | controller | - ' ' 2026-03-09 19:10:03.741732 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.741737 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.741741 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.741745 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.741750 | controller | --timeout=300s' 2026-03-09 19:10:03.741754 | controller | - ' ' 2026-03-09 19:10:03.741758 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.741762 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.741767 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.741771 | controller | - ' ' 2026-03-09 19:10:03.741775 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741780 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741784 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.741788 | controller | - ' ' 2026-03-09 19:10:03.741792 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.741797 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.741801 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.741805 | controller | available"' 2026-03-09 19:10:03.741809 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.741816 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.741821 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.741825 | controller | - ' exit 1' 2026-03-09 19:10:03.741829 | controller | - ' fi' 2026-03-09 19:10:03.741834 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.741838 | controller | - ' ' 2026-03-09 19:10:03.741842 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.741846 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.741851 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741855 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.741859 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741864 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.741868 | controller | -- \' 2026-03-09 19:10:03.741872 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.741876 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.741881 | controller | - ' logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.741887 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.741891 | controller | - ' logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.741895 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.741900 | controller | found' 2026-03-09 19:10:03.741904 | controller | - ' logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.741915 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.741920 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.741924 | controller | - ' ' 2026-03-09 19:10:03.741928 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.741933 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.741937 | controller | - ' ' 2026-03-09 19:10:03.741941 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.741945 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.741950 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.741954 | controller | - ' exit 1' 2026-03-09 19:10:03.741958 | controller | - ' fi' 2026-03-09 19:10:03.741962 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.741967 | controller | - ' ' 2026-03-09 19:10:03.741971 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.741975 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.741980 | controller | | base64 -d)' 2026-03-09 19:10:03.741984 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.741988 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.741992 | controller | - ' exit 1' 2026-03-09 19:10:03.741997 | controller | - ' fi' 2026-03-09 19:10:03.742001 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.742006 | controller | - ' ' 2026-03-09 19:10:03.742012 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.742018 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.742024 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.742030 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.742036 | controller | --timeout=300s' 2026-03-09 19:10:03.742041 | controller | - ' ' 2026-03-09 19:10:03.742045 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.742050 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.742054 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.742058 | controller | - ' ' 2026-03-09 19:10:03.742065 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742069 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742073 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742078 | controller | - ' ' 2026-03-09 19:10:03.742082 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.742086 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.742090 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.742097 | controller | available"' 2026-03-09 19:10:03.742101 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.742105 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.742110 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.742114 | controller | - ' exit 1' 2026-03-09 19:10:03.742118 | controller | - ' fi' 2026-03-09 19:10:03.742122 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.742127 | controller | - ' ' 2026-03-09 19:10:03.742131 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.742135 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.742139 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742144 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.742162 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742166 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.742171 | controller | -- \' 2026-03-09 19:10:03.742175 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742179 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.742184 | controller | - ' logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.742188 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.742192 | controller | - ' logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.742196 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.742201 | controller | found' 2026-03-09 19:10:03.742205 | controller | - ' logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.742209 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.742214 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.742226 | controller | - ' ' 2026-03-09 19:10:03.742231 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.742235 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.742240 | controller | - ' ' 2026-03-09 19:10:03.742244 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.742248 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.742253 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.742257 | controller | - ' exit 1' 2026-03-09 19:10:03.742261 | controller | - ' fi' 2026-03-09 19:10:03.742265 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.742270 | controller | - ' ' 2026-03-09 19:10:03.742274 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.742278 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.742282 | controller | | base64 -d)' 2026-03-09 19:10:03.742287 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.742291 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.742295 | controller | - ' exit 1' 2026-03-09 19:10:03.742300 | controller | - ' fi' 2026-03-09 19:10:03.742304 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.742308 | controller | - ' ' 2026-03-09 19:10:03.742312 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.742319 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.742323 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.742327 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.742332 | controller | --timeout=300s' 2026-03-09 19:10:03.742336 | controller | - ' ' 2026-03-09 19:10:03.742340 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.742345 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.742349 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.742353 | controller | - ' ' 2026-03-09 19:10:03.742357 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742362 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742366 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742370 | controller | - ' ' 2026-03-09 19:10:03.742375 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.742379 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.742383 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.742388 | controller | available"' 2026-03-09 19:10:03.742394 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.742398 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.742402 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.742407 | controller | - ' exit 1' 2026-03-09 19:10:03.742411 | controller | - ' fi' 2026-03-09 19:10:03.742415 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.742420 | controller | - ' ' 2026-03-09 19:10:03.742425 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.742430 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.742434 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742438 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.742445 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742449 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.742453 | controller | -- \' 2026-03-09 19:10:03.742458 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742462 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.742466 | controller | - ' logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.742470 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.742475 | controller | - ' logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.742479 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.742483 | controller | found' 2026-03-09 19:10:03.742487 | controller | - ' logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.742492 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.742498 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.742502 | controller | - ' ' 2026-03-09 19:10:03.742513 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.742518 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.742522 | controller | - ' ' 2026-03-09 19:10:03.742527 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.742531 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.742535 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.742540 | controller | - ' exit 1' 2026-03-09 19:10:03.742544 | controller | - ' fi' 2026-03-09 19:10:03.742548 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.742552 | controller | - ' ' 2026-03-09 19:10:03.742557 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.742561 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.742565 | controller | | base64 -d)' 2026-03-09 19:10:03.742570 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.742574 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.742578 | controller | - ' exit 1' 2026-03-09 19:10:03.742582 | controller | - ' fi' 2026-03-09 19:10:03.742587 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.742591 | controller | - ' ' 2026-03-09 19:10:03.742595 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.742600 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.742604 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.742608 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.742612 | controller | --timeout=300s' 2026-03-09 19:10:03.742617 | controller | - ' ' 2026-03-09 19:10:03.742621 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.742625 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.742630 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.742634 | controller | - ' ' 2026-03-09 19:10:03.742638 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742642 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742647 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742651 | controller | - ' ' 2026-03-09 19:10:03.742655 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.742659 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.742664 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.742668 | controller | available"' 2026-03-09 19:10:03.742672 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.742678 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.742682 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.742687 | controller | - ' exit 1' 2026-03-09 19:10:03.742691 | controller | - ' fi' 2026-03-09 19:10:03.742695 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.742700 | controller | - ' ' 2026-03-09 19:10:03.742708 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.742713 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.742717 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742721 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.742725 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742730 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.742734 | controller | -- \' 2026-03-09 19:10:03.742738 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.742742 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.742747 | controller | - ' logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.742751 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.742755 | controller | - ' logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.742759 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.742764 | controller | found' 2026-03-09 19:10:03.742768 | controller | - ' logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.742772 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.742776 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.742781 | controller | - ' ' 2026-03-09 19:10:03.742785 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.742796 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.742800 | controller | - ' ' 2026-03-09 19:10:03.742805 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.742809 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.742813 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.742818 | controller | - ' exit 1' 2026-03-09 19:10:03.742822 | controller | - ' fi' 2026-03-09 19:10:03.742826 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.742831 | controller | - ' ' 2026-03-09 19:10:03.742835 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.742839 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.742843 | controller | | base64 -d)' 2026-03-09 19:10:03.742848 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.742852 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.742856 | controller | - ' exit 1' 2026-03-09 19:10:03.742861 | controller | - ' fi' 2026-03-09 19:10:03.742865 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.742869 | controller | - ' ' 2026-03-09 19:10:03.742873 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.742878 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.742884 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.742890 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.742896 | controller | --timeout=300s' 2026-03-09 19:10:03.742902 | controller | - ' ' 2026-03-09 19:10:03.742908 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.742912 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.742919 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.742924 | controller | - ' ' 2026-03-09 19:10:03.742928 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742932 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742936 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.742941 | controller | - ' ' 2026-03-09 19:10:03.742945 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.742949 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.742953 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.742958 | controller | available"' 2026-03-09 19:10:03.742962 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.742966 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.742970 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.742975 | controller | - ' exit 1' 2026-03-09 19:10:03.742979 | controller | - ' fi' 2026-03-09 19:10:03.742983 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.742987 | controller | - ' ' 2026-03-09 19:10:03.742992 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.742996 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.743000 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743007 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.743014 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743020 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.743025 | controller | -- \' 2026-03-09 19:10:03.743031 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743037 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.743043 | controller | - ' logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.743048 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.743053 | controller | - ' logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.743057 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.743061 | controller | found' 2026-03-09 19:10:03.743066 | controller | - ' logger.go:42: 18:55:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.743070 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.743074 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.743078 | controller | - ' ' 2026-03-09 19:10:03.743083 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.743095 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.743100 | controller | - ' ' 2026-03-09 19:10:03.743104 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.743108 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.743113 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.743119 | controller | - ' exit 1' 2026-03-09 19:10:03.743124 | controller | - ' fi' 2026-03-09 19:10:03.743128 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.743132 | controller | - ' ' 2026-03-09 19:10:03.743137 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.743141 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.743157 | controller | | base64 -d)' 2026-03-09 19:10:03.743164 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.743168 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.743173 | controller | - ' exit 1' 2026-03-09 19:10:03.743177 | controller | - ' fi' 2026-03-09 19:10:03.743181 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.743185 | controller | - ' ' 2026-03-09 19:10:03.743190 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.743194 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.743198 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.743202 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.743207 | controller | --timeout=300s' 2026-03-09 19:10:03.743211 | controller | - ' ' 2026-03-09 19:10:03.743215 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.743220 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.743224 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.743228 | controller | - ' ' 2026-03-09 19:10:03.743232 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743237 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743241 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743245 | controller | - ' ' 2026-03-09 19:10:03.743249 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.743254 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.743258 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.743262 | controller | available"' 2026-03-09 19:10:03.743267 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.743271 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.743275 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.743279 | controller | - ' exit 1' 2026-03-09 19:10:03.743284 | controller | - ' fi' 2026-03-09 19:10:03.743288 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.743292 | controller | - ' ' 2026-03-09 19:10:03.743296 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.743301 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.743305 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743309 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.743314 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743320 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.743324 | controller | -- \' 2026-03-09 19:10:03.743328 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743333 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.743337 | controller | - ' logger.go:42: 18:55:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.743341 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.743348 | controller | - ' logger.go:42: 18:55:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.743352 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.743357 | controller | found' 2026-03-09 19:10:03.743361 | controller | - ' logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.743366 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.743370 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.743374 | controller | - ' ' 2026-03-09 19:10:03.743379 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.743383 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.743387 | controller | - ' ' 2026-03-09 19:10:03.743399 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.743404 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.743408 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.743413 | controller | - ' exit 1' 2026-03-09 19:10:03.743417 | controller | - ' fi' 2026-03-09 19:10:03.743421 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.743425 | controller | - ' ' 2026-03-09 19:10:03.743430 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.743434 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.743438 | controller | | base64 -d)' 2026-03-09 19:10:03.743442 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.743449 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.743453 | controller | - ' exit 1' 2026-03-09 19:10:03.743458 | controller | - ' fi' 2026-03-09 19:10:03.743462 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.743466 | controller | - ' ' 2026-03-09 19:10:03.743472 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.743476 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.743481 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.743485 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.743489 | controller | --timeout=300s' 2026-03-09 19:10:03.743494 | controller | - ' ' 2026-03-09 19:10:03.743498 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.743502 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.743507 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.743511 | controller | - ' ' 2026-03-09 19:10:03.743515 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743521 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743527 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743532 | controller | - ' ' 2026-03-09 19:10:03.743536 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.743540 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.743545 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.743549 | controller | available"' 2026-03-09 19:10:03.743553 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.743557 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.743562 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.743566 | controller | - ' exit 1' 2026-03-09 19:10:03.743570 | controller | - ' fi' 2026-03-09 19:10:03.743575 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.743579 | controller | - ' ' 2026-03-09 19:10:03.743583 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.743587 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.743592 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743596 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.743600 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743605 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.743609 | controller | -- \' 2026-03-09 19:10:03.743613 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743619 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.743624 | controller | - ' logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.743628 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.743633 | controller | - ' logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.743637 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.743641 | controller | found' 2026-03-09 19:10:03.743645 | controller | - ' logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.743650 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.743654 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.743658 | controller | - ' ' 2026-03-09 19:10:03.743663 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.743667 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.743671 | controller | - ' ' 2026-03-09 19:10:03.743695 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.743701 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.743706 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.743710 | controller | - ' exit 1' 2026-03-09 19:10:03.743714 | controller | - ' fi' 2026-03-09 19:10:03.743719 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.743723 | controller | - ' ' 2026-03-09 19:10:03.743727 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.743731 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.743735 | controller | | base64 -d)' 2026-03-09 19:10:03.743740 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.743746 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.743751 | controller | - ' exit 1' 2026-03-09 19:10:03.743755 | controller | - ' fi' 2026-03-09 19:10:03.743759 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.743763 | controller | - ' ' 2026-03-09 19:10:03.743768 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.743772 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.743776 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.743781 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.743785 | controller | --timeout=300s' 2026-03-09 19:10:03.743789 | controller | - ' ' 2026-03-09 19:10:03.743793 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.743798 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.743802 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.743806 | controller | - ' ' 2026-03-09 19:10:03.743811 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743815 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743819 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.743823 | controller | - ' ' 2026-03-09 19:10:03.743828 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.743832 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.743836 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.743841 | controller | available"' 2026-03-09 19:10:03.743845 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.743849 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.743853 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.743858 | controller | - ' exit 1' 2026-03-09 19:10:03.743862 | controller | - ' fi' 2026-03-09 19:10:03.743866 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.743870 | controller | - ' ' 2026-03-09 19:10:03.743875 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.743879 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.743883 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743887 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.743892 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743896 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.743900 | controller | -- \' 2026-03-09 19:10:03.743905 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.743909 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.743913 | controller | - ' logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.743919 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.743924 | controller | - ' logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.743928 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.743932 | controller | found' 2026-03-09 19:10:03.743937 | controller | - ' logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.743941 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.743945 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.743950 | controller | - ' ' 2026-03-09 19:10:03.743954 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.743958 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.743962 | controller | - ' ' 2026-03-09 19:10:03.743967 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.743978 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.743982 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.743987 | controller | - ' exit 1' 2026-03-09 19:10:03.743991 | controller | - ' fi' 2026-03-09 19:10:03.743995 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.744000 | controller | - ' ' 2026-03-09 19:10:03.744007 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.744013 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.744019 | controller | | base64 -d)' 2026-03-09 19:10:03.744025 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.744031 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.744036 | controller | - ' exit 1' 2026-03-09 19:10:03.744042 | controller | - ' fi' 2026-03-09 19:10:03.744047 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.744053 | controller | - ' ' 2026-03-09 19:10:03.744059 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.744064 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.744070 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.744075 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.744081 | controller | --timeout=300s' 2026-03-09 19:10:03.744086 | controller | - ' ' 2026-03-09 19:10:03.744091 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.744097 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.744103 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.744108 | controller | - ' ' 2026-03-09 19:10:03.744113 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744118 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744122 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744126 | controller | - ' ' 2026-03-09 19:10:03.744130 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.744135 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.744139 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.744156 | controller | available"' 2026-03-09 19:10:03.744163 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.744171 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.744175 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.744180 | controller | - ' exit 1' 2026-03-09 19:10:03.744184 | controller | - ' fi' 2026-03-09 19:10:03.744188 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.744193 | controller | - ' ' 2026-03-09 19:10:03.744197 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.744201 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.744205 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744210 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.744214 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744218 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.744223 | controller | -- \' 2026-03-09 19:10:03.744227 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744231 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.744235 | controller | - ' logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.744240 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.744248 | controller | - ' logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.744252 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.744256 | controller | found' 2026-03-09 19:10:03.744260 | controller | - ' logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.744265 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.744269 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.744273 | controller | - ' ' 2026-03-09 19:10:03.744278 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.744282 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.744286 | controller | - ' ' 2026-03-09 19:10:03.744290 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.744295 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.744307 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.744312 | controller | - ' exit 1' 2026-03-09 19:10:03.744316 | controller | - ' fi' 2026-03-09 19:10:03.744321 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.744325 | controller | - ' ' 2026-03-09 19:10:03.744329 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.744334 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.744338 | controller | | base64 -d)' 2026-03-09 19:10:03.744342 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.744346 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.744351 | controller | - ' exit 1' 2026-03-09 19:10:03.744355 | controller | - ' fi' 2026-03-09 19:10:03.744359 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.744364 | controller | - ' ' 2026-03-09 19:10:03.744368 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.744372 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.744378 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.744382 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.744387 | controller | --timeout=300s' 2026-03-09 19:10:03.744391 | controller | - ' ' 2026-03-09 19:10:03.744395 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.744400 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.744404 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.744408 | controller | - ' ' 2026-03-09 19:10:03.744413 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744417 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744421 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744425 | controller | - ' ' 2026-03-09 19:10:03.744430 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.744434 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.744438 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.744442 | controller | available"' 2026-03-09 19:10:03.744447 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.744451 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.744455 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.744459 | controller | - ' exit 1' 2026-03-09 19:10:03.744464 | controller | - ' fi' 2026-03-09 19:10:03.744468 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.744472 | controller | - ' ' 2026-03-09 19:10:03.744476 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.744481 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.744485 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744489 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.744494 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744498 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.744504 | controller | -- \' 2026-03-09 19:10:03.744508 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744512 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.744517 | controller | - ' logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.744521 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.744525 | controller | - ' logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.744529 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.744534 | controller | found' 2026-03-09 19:10:03.744538 | controller | - ' logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.744542 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.744546 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.744551 | controller | - ' ' 2026-03-09 19:10:03.744557 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.744561 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.744565 | controller | - ' ' 2026-03-09 19:10:03.744569 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.744574 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.744578 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.744589 | controller | - ' exit 1' 2026-03-09 19:10:03.744594 | controller | - ' fi' 2026-03-09 19:10:03.744600 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.744605 | controller | - ' ' 2026-03-09 19:10:03.744609 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.744613 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.744617 | controller | | base64 -d)' 2026-03-09 19:10:03.744622 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.744626 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.744630 | controller | - ' exit 1' 2026-03-09 19:10:03.744635 | controller | - ' fi' 2026-03-09 19:10:03.744640 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.744647 | controller | - ' ' 2026-03-09 19:10:03.744652 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.744658 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.744664 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.744670 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.744676 | controller | --timeout=300s' 2026-03-09 19:10:03.744681 | controller | - ' ' 2026-03-09 19:10:03.744688 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.744694 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.744701 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.744707 | controller | - ' ' 2026-03-09 19:10:03.744713 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744719 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744725 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.744731 | controller | - ' ' 2026-03-09 19:10:03.744737 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.744741 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.744745 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.744750 | controller | available"' 2026-03-09 19:10:03.744754 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.744758 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.744762 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.744767 | controller | - ' exit 1' 2026-03-09 19:10:03.744771 | controller | - ' fi' 2026-03-09 19:10:03.744775 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.744779 | controller | - ' ' 2026-03-09 19:10:03.744784 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.744791 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.744795 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744799 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.744803 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744808 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.744812 | controller | -- \' 2026-03-09 19:10:03.744816 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.744821 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.744827 | controller | - ' logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.744831 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.744836 | controller | - ' logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.744842 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.744846 | controller | found' 2026-03-09 19:10:03.744850 | controller | - ' logger.go:42: 18:55:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.744855 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.744859 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.744863 | controller | - ' ' 2026-03-09 19:10:03.744867 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.744872 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.744876 | controller | - ' ' 2026-03-09 19:10:03.744880 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.744884 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.744889 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.744893 | controller | - ' exit 1' 2026-03-09 19:10:03.744897 | controller | - ' fi' 2026-03-09 19:10:03.744911 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.744915 | controller | - ' ' 2026-03-09 19:10:03.744920 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.744924 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.744928 | controller | | base64 -d)' 2026-03-09 19:10:03.744933 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.744937 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.744941 | controller | - ' exit 1' 2026-03-09 19:10:03.744945 | controller | - ' fi' 2026-03-09 19:10:03.744950 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.744954 | controller | - ' ' 2026-03-09 19:10:03.744958 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.744963 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.744967 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.744971 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.744975 | controller | --timeout=300s' 2026-03-09 19:10:03.744981 | controller | - ' ' 2026-03-09 19:10:03.744986 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.744990 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.744997 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.745002 | controller | - ' ' 2026-03-09 19:10:03.745007 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745013 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745019 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745025 | controller | - ' ' 2026-03-09 19:10:03.745031 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.745037 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.745042 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.745046 | controller | available"' 2026-03-09 19:10:03.745051 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.745055 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.745059 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.745063 | controller | - ' exit 1' 2026-03-09 19:10:03.745068 | controller | - ' fi' 2026-03-09 19:10:03.745072 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.745076 | controller | - ' ' 2026-03-09 19:10:03.745080 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.745085 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.745089 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745093 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.745098 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745102 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.745106 | controller | -- \' 2026-03-09 19:10:03.745110 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745115 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.745119 | controller | - ' logger.go:42: 18:55:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.745123 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.745128 | controller | - ' logger.go:42: 18:55:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.745132 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.745136 | controller | found' 2026-03-09 19:10:03.745140 | controller | - ' logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.745156 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.745287 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.745293 | controller | - ' ' 2026-03-09 19:10:03.745298 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.745302 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.745306 | controller | - ' ' 2026-03-09 19:10:03.745310 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.745315 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.745319 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.745326 | controller | - ' exit 1' 2026-03-09 19:10:03.745330 | controller | - ' fi' 2026-03-09 19:10:03.745334 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.745339 | controller | - ' ' 2026-03-09 19:10:03.745353 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.745357 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.745362 | controller | | base64 -d)' 2026-03-09 19:10:03.745366 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.745370 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.745375 | controller | - ' exit 1' 2026-03-09 19:10:03.745379 | controller | - ' fi' 2026-03-09 19:10:03.745383 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.745387 | controller | - ' ' 2026-03-09 19:10:03.745391 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.745396 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.745400 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.745404 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.745408 | controller | --timeout=300s' 2026-03-09 19:10:03.745413 | controller | - ' ' 2026-03-09 19:10:03.745417 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.745421 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.745425 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.745430 | controller | - ' ' 2026-03-09 19:10:03.745434 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745438 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745442 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745447 | controller | - ' ' 2026-03-09 19:10:03.745451 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.745455 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.745459 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.745463 | controller | available"' 2026-03-09 19:10:03.745468 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.745474 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.745478 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.745482 | controller | - ' exit 1' 2026-03-09 19:10:03.745487 | controller | - ' fi' 2026-03-09 19:10:03.745491 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.745495 | controller | - ' ' 2026-03-09 19:10:03.745499 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.745504 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.745508 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745512 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.745516 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745520 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.745527 | controller | -- \' 2026-03-09 19:10:03.745531 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745535 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.745540 | controller | - ' logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.745544 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.745548 | controller | - ' logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.745552 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.745557 | controller | found' 2026-03-09 19:10:03.745562 | controller | - ' logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.745567 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.745571 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.745575 | controller | - ' ' 2026-03-09 19:10:03.745580 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.745584 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.745588 | controller | - ' ' 2026-03-09 19:10:03.745592 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.745597 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.745601 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.745605 | controller | - ' exit 1' 2026-03-09 19:10:03.745609 | controller | - ' fi' 2026-03-09 19:10:03.745614 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.745618 | controller | - ' ' 2026-03-09 19:10:03.745630 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.745634 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.745639 | controller | | base64 -d)' 2026-03-09 19:10:03.745643 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.745647 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.745652 | controller | - ' exit 1' 2026-03-09 19:10:03.745656 | controller | - ' fi' 2026-03-09 19:10:03.745660 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.745664 | controller | - ' ' 2026-03-09 19:10:03.745669 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.745673 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.745677 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.745681 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.745685 | controller | --timeout=300s' 2026-03-09 19:10:03.745690 | controller | - ' ' 2026-03-09 19:10:03.745694 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.745698 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.745703 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.745707 | controller | - ' ' 2026-03-09 19:10:03.745711 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745715 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745723 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.745730 | controller | - ' ' 2026-03-09 19:10:03.745734 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.745739 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.745743 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.745747 | controller | available"' 2026-03-09 19:10:03.745751 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.745756 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.745760 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.745764 | controller | - ' exit 1' 2026-03-09 19:10:03.745768 | controller | - ' fi' 2026-03-09 19:10:03.745775 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.745779 | controller | - ' ' 2026-03-09 19:10:03.745783 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.745787 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.745792 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745796 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.745800 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745805 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.745809 | controller | -- \' 2026-03-09 19:10:03.745813 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.745817 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.745821 | controller | - ' logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.745826 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.745830 | controller | - ' logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.745834 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.745838 | controller | found' 2026-03-09 19:10:03.745843 | controller | - ' logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.745847 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.745851 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.745855 | controller | - ' ' 2026-03-09 19:10:03.745860 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.745864 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.745868 | controller | - ' ' 2026-03-09 19:10:03.745872 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.745876 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.745881 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.745885 | controller | - ' exit 1' 2026-03-09 19:10:03.745889 | controller | - ' fi' 2026-03-09 19:10:03.745893 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.745898 | controller | - ' ' 2026-03-09 19:10:03.745902 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.745914 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.745918 | controller | | base64 -d)' 2026-03-09 19:10:03.745922 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.745929 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.745933 | controller | - ' exit 1' 2026-03-09 19:10:03.745938 | controller | - ' fi' 2026-03-09 19:10:03.745942 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.745946 | controller | - ' ' 2026-03-09 19:10:03.745952 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.745958 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.745963 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.745967 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.745971 | controller | --timeout=300s' 2026-03-09 19:10:03.745976 | controller | - ' ' 2026-03-09 19:10:03.745980 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.745984 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.745989 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.745993 | controller | - ' ' 2026-03-09 19:10:03.745997 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746001 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746006 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746012 | controller | - ' ' 2026-03-09 19:10:03.746018 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.746024 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.746031 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.746037 | controller | available"' 2026-03-09 19:10:03.746042 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.746046 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.746051 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.746055 | controller | - ' exit 1' 2026-03-09 19:10:03.746059 | controller | - ' fi' 2026-03-09 19:10:03.746063 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.746068 | controller | - ' ' 2026-03-09 19:10:03.746072 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.746076 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.746080 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746085 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.746089 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746093 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.746097 | controller | -- \' 2026-03-09 19:10:03.746102 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746106 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.746111 | controller | - ' logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.746116 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.746124 | controller | - ' logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.746128 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.746133 | controller | found' 2026-03-09 19:10:03.746137 | controller | - ' logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.746141 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.746158 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.746165 | controller | - ' ' 2026-03-09 19:10:03.746172 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.746176 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.746180 | controller | - ' ' 2026-03-09 19:10:03.746184 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.746189 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.746193 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.746197 | controller | - ' exit 1' 2026-03-09 19:10:03.746201 | controller | - ' fi' 2026-03-09 19:10:03.746206 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.746210 | controller | - ' ' 2026-03-09 19:10:03.746214 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.746227 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.746232 | controller | | base64 -d)' 2026-03-09 19:10:03.746236 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.746240 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.746245 | controller | - ' exit 1' 2026-03-09 19:10:03.746249 | controller | - ' fi' 2026-03-09 19:10:03.746253 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.746258 | controller | - ' ' 2026-03-09 19:10:03.746262 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.746266 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.746270 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.746275 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.746279 | controller | --timeout=300s' 2026-03-09 19:10:03.746283 | controller | - ' ' 2026-03-09 19:10:03.746288 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.746292 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.746296 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.746300 | controller | - ' ' 2026-03-09 19:10:03.746306 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746311 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746315 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746319 | controller | - ' ' 2026-03-09 19:10:03.746323 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.746328 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.746332 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.746336 | controller | available"' 2026-03-09 19:10:03.746340 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.746348 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.746353 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.746360 | controller | - ' exit 1' 2026-03-09 19:10:03.746365 | controller | - ' fi' 2026-03-09 19:10:03.746371 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.746377 | controller | - ' ' 2026-03-09 19:10:03.746383 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.746389 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.746396 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746401 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.746407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746413 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.746418 | controller | -- \' 2026-03-09 19:10:03.746424 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746430 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.746436 | controller | - ' logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.746441 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.746447 | controller | - ' logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.746453 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.746459 | controller | found' 2026-03-09 19:10:03.746464 | controller | - ' logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.746470 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.746476 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.746482 | controller | - ' ' 2026-03-09 19:10:03.746488 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.746494 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.746499 | controller | - ' ' 2026-03-09 19:10:03.746503 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.746507 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.746512 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.746516 | controller | - ' exit 1' 2026-03-09 19:10:03.746520 | controller | - ' fi' 2026-03-09 19:10:03.746524 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.746529 | controller | - ' ' 2026-03-09 19:10:03.746533 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.746537 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.746541 | controller | | base64 -d)' 2026-03-09 19:10:03.746556 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.746560 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.746565 | controller | - ' exit 1' 2026-03-09 19:10:03.746569 | controller | - ' fi' 2026-03-09 19:10:03.746573 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.746578 | controller | - ' ' 2026-03-09 19:10:03.746582 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.746586 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.746590 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.746598 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.746602 | controller | --timeout=300s' 2026-03-09 19:10:03.746606 | controller | - ' ' 2026-03-09 19:10:03.746611 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.746615 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.746619 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.746623 | controller | - ' ' 2026-03-09 19:10:03.746628 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746632 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746636 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746640 | controller | - ' ' 2026-03-09 19:10:03.746645 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.746649 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.746653 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.746658 | controller | available"' 2026-03-09 19:10:03.746662 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.746666 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.746670 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.746675 | controller | - ' exit 1' 2026-03-09 19:10:03.746679 | controller | - ' fi' 2026-03-09 19:10:03.746683 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.746688 | controller | - ' ' 2026-03-09 19:10:03.746692 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.746696 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.746700 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746705 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.746709 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746713 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.746718 | controller | -- \' 2026-03-09 19:10:03.746722 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746726 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.746734 | controller | - ' logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.746738 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.746743 | controller | - ' logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.746747 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.746751 | controller | found' 2026-03-09 19:10:03.746758 | controller | - ' logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.746762 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.746767 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.746771 | controller | - ' ' 2026-03-09 19:10:03.746775 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.746781 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.746787 | controller | - ' ' 2026-03-09 19:10:03.746792 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.746796 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.746800 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.746805 | controller | - ' exit 1' 2026-03-09 19:10:03.746809 | controller | - ' fi' 2026-03-09 19:10:03.746813 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.746817 | controller | - ' ' 2026-03-09 19:10:03.746822 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.746826 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.746830 | controller | | base64 -d)' 2026-03-09 19:10:03.746835 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.746847 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.746853 | controller | - ' exit 1' 2026-03-09 19:10:03.746857 | controller | - ' fi' 2026-03-09 19:10:03.746862 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.746866 | controller | - ' ' 2026-03-09 19:10:03.746870 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.746875 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.746879 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.746883 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.746887 | controller | --timeout=300s' 2026-03-09 19:10:03.746892 | controller | - ' ' 2026-03-09 19:10:03.746896 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.746901 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.746906 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.746911 | controller | - ' ' 2026-03-09 19:10:03.746915 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746920 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746924 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.746928 | controller | - ' ' 2026-03-09 19:10:03.746932 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.746937 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.746941 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.746945 | controller | available"' 2026-03-09 19:10:03.746949 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.746954 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.746958 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.746962 | controller | - ' exit 1' 2026-03-09 19:10:03.746966 | controller | - ' fi' 2026-03-09 19:10:03.746971 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.746975 | controller | - ' ' 2026-03-09 19:10:03.746979 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.746983 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.746989 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.746994 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.746998 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747002 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.747008 | controller | -- \' 2026-03-09 19:10:03.747014 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747020 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.747025 | controller | - ' logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.747032 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.747038 | controller | - ' logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.747043 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.747047 | controller | found' 2026-03-09 19:10:03.747051 | controller | - ' logger.go:42: 18:55:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.747055 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.747060 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.747068 | controller | - ' ' 2026-03-09 19:10:03.747072 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.747077 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.747081 | controller | - ' ' 2026-03-09 19:10:03.747085 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.747090 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.747094 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.747099 | controller | - ' exit 1' 2026-03-09 19:10:03.747103 | controller | - ' fi' 2026-03-09 19:10:03.747108 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.747113 | controller | - ' ' 2026-03-09 19:10:03.747119 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.747123 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.747128 | controller | | base64 -d)' 2026-03-09 19:10:03.747133 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.747159 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.747167 | controller | - ' exit 1' 2026-03-09 19:10:03.747171 | controller | - ' fi' 2026-03-09 19:10:03.747176 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.747180 | controller | - ' ' 2026-03-09 19:10:03.747184 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.747189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.747193 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.747197 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.747202 | controller | --timeout=300s' 2026-03-09 19:10:03.747206 | controller | - ' ' 2026-03-09 19:10:03.747210 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.747215 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.747219 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.747225 | controller | - ' ' 2026-03-09 19:10:03.747230 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747234 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747238 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747243 | controller | - ' ' 2026-03-09 19:10:03.747247 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.747251 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.747256 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.747260 | controller | available"' 2026-03-09 19:10:03.747264 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.747269 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.747273 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.747277 | controller | - ' exit 1' 2026-03-09 19:10:03.747282 | controller | - ' fi' 2026-03-09 19:10:03.747286 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.747290 | controller | - ' ' 2026-03-09 19:10:03.747295 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.747299 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.747303 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747307 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.747312 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747316 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.747320 | controller | -- \' 2026-03-09 19:10:03.747325 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747329 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.747333 | controller | - ' logger.go:42: 18:55:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.747338 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.747342 | controller | - ' logger.go:42: 18:55:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.747346 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.747351 | controller | found' 2026-03-09 19:10:03.747355 | controller | - ' logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.747359 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.747364 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.747368 | controller | - ' ' 2026-03-09 19:10:03.747372 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.747376 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.747381 | controller | - ' ' 2026-03-09 19:10:03.747387 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.747391 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.747395 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.747400 | controller | - ' exit 1' 2026-03-09 19:10:03.747404 | controller | - ' fi' 2026-03-09 19:10:03.747410 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.747414 | controller | - ' ' 2026-03-09 19:10:03.747419 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.747423 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.747427 | controller | | base64 -d)' 2026-03-09 19:10:03.747431 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.747436 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.747448 | controller | - ' exit 1' 2026-03-09 19:10:03.747452 | controller | - ' fi' 2026-03-09 19:10:03.747457 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.747461 | controller | - ' ' 2026-03-09 19:10:03.747466 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.747470 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.747474 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.747478 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.747483 | controller | --timeout=300s' 2026-03-09 19:10:03.747487 | controller | - ' ' 2026-03-09 19:10:03.747491 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.747496 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.747500 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.747504 | controller | - ' ' 2026-03-09 19:10:03.747508 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747513 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747517 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747521 | controller | - ' ' 2026-03-09 19:10:03.747525 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.747530 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.747534 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.747538 | controller | available"' 2026-03-09 19:10:03.747543 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.747547 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.747551 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.747555 | controller | - ' exit 1' 2026-03-09 19:10:03.747560 | controller | - ' fi' 2026-03-09 19:10:03.747564 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.747568 | controller | - ' ' 2026-03-09 19:10:03.747572 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.747577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.747581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747585 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.747590 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747594 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.747600 | controller | -- \' 2026-03-09 19:10:03.747623 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747627 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.747631 | controller | - ' logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.747636 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.747640 | controller | - ' logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.747644 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.747648 | controller | found' 2026-03-09 19:10:03.747653 | controller | - ' logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.747657 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.747661 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.747666 | controller | - ' ' 2026-03-09 19:10:03.747670 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.747674 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.747694 | controller | - ' ' 2026-03-09 19:10:03.747699 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.747704 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.747708 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.747712 | controller | - ' exit 1' 2026-03-09 19:10:03.747716 | controller | - ' fi' 2026-03-09 19:10:03.747721 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.747725 | controller | - ' ' 2026-03-09 19:10:03.747729 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.747734 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.747738 | controller | | base64 -d)' 2026-03-09 19:10:03.747742 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.747750 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.747754 | controller | - ' exit 1' 2026-03-09 19:10:03.747759 | controller | - ' fi' 2026-03-09 19:10:03.747771 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.747776 | controller | - ' ' 2026-03-09 19:10:03.747780 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.747784 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.747789 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.747793 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.747797 | controller | --timeout=300s' 2026-03-09 19:10:03.747802 | controller | - ' ' 2026-03-09 19:10:03.747806 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.747810 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.747815 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.747819 | controller | - ' ' 2026-03-09 19:10:03.747823 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747827 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747832 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.747838 | controller | - ' ' 2026-03-09 19:10:03.747842 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.747848 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.747853 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.747857 | controller | available"' 2026-03-09 19:10:03.747861 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.747865 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.747870 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.747874 | controller | - ' exit 1' 2026-03-09 19:10:03.747878 | controller | - ' fi' 2026-03-09 19:10:03.747882 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.747887 | controller | - ' ' 2026-03-09 19:10:03.747891 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.747895 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.747900 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747904 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.747908 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747912 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.747917 | controller | -- \' 2026-03-09 19:10:03.747921 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.747925 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.747930 | controller | - ' logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.747934 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.747938 | controller | - ' logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.747942 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.747947 | controller | found' 2026-03-09 19:10:03.747951 | controller | - ' logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.747955 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.747959 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.747964 | controller | - ' ' 2026-03-09 19:10:03.747968 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.747972 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.747977 | controller | - ' ' 2026-03-09 19:10:03.747981 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.747985 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.747989 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.747994 | controller | - ' exit 1' 2026-03-09 19:10:03.747998 | controller | - ' fi' 2026-03-09 19:10:03.748002 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.748016 | controller | - ' ' 2026-03-09 19:10:03.748023 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.748029 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.748035 | controller | | base64 -d)' 2026-03-09 19:10:03.748041 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.748049 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.748058 | controller | - ' exit 1' 2026-03-09 19:10:03.748064 | controller | - ' fi' 2026-03-09 19:10:03.748069 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.748075 | controller | - ' ' 2026-03-09 19:10:03.748090 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.748097 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.748103 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.748108 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.748112 | controller | --timeout=300s' 2026-03-09 19:10:03.748117 | controller | - ' ' 2026-03-09 19:10:03.748121 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.748125 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.748129 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.748134 | controller | - ' ' 2026-03-09 19:10:03.748138 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748142 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748163 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748167 | controller | - ' ' 2026-03-09 19:10:03.748172 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.748176 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.748180 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.748184 | controller | available"' 2026-03-09 19:10:03.748189 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.748195 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.748200 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.748204 | controller | - ' exit 1' 2026-03-09 19:10:03.748208 | controller | - ' fi' 2026-03-09 19:10:03.748212 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.748217 | controller | - ' ' 2026-03-09 19:10:03.748221 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.748225 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.748229 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748234 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.748238 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748242 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.748247 | controller | -- \' 2026-03-09 19:10:03.748251 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748255 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.748259 | controller | - ' logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.748264 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.748268 | controller | - ' logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.748275 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.748279 | controller | found' 2026-03-09 19:10:03.748284 | controller | - ' logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.748288 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.748292 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.748296 | controller | - ' ' 2026-03-09 19:10:03.748301 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.748305 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.748309 | controller | - ' ' 2026-03-09 19:10:03.748314 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.748318 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.748322 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.748326 | controller | - ' exit 1' 2026-03-09 19:10:03.748331 | controller | - ' fi' 2026-03-09 19:10:03.748336 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.748342 | controller | - ' ' 2026-03-09 19:10:03.748348 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.748354 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.748360 | controller | | base64 -d)' 2026-03-09 19:10:03.748366 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.748372 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.748378 | controller | - ' exit 1' 2026-03-09 19:10:03.748384 | controller | - ' fi' 2026-03-09 19:10:03.748390 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.748397 | controller | - ' ' 2026-03-09 19:10:03.748414 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.748421 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.748427 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.748433 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.748437 | controller | --timeout=300s' 2026-03-09 19:10:03.748441 | controller | - ' ' 2026-03-09 19:10:03.748446 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.748450 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.748454 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.748459 | controller | - ' ' 2026-03-09 19:10:03.748463 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748467 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748471 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748476 | controller | - ' ' 2026-03-09 19:10:03.748480 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.748484 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.748488 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.748493 | controller | available"' 2026-03-09 19:10:03.748497 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.748501 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.748508 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.748513 | controller | - ' exit 1' 2026-03-09 19:10:03.748517 | controller | - ' fi' 2026-03-09 19:10:03.748521 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.748525 | controller | - ' ' 2026-03-09 19:10:03.748530 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.748534 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.748538 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748542 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.748547 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748551 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.748555 | controller | -- \' 2026-03-09 19:10:03.748559 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748564 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.748568 | controller | - ' logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.748572 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.748577 | controller | - ' logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.748581 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.748585 | controller | found' 2026-03-09 19:10:03.748589 | controller | - ' logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.748594 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.748598 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.748602 | controller | - ' ' 2026-03-09 19:10:03.748606 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.748611 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.748615 | controller | - ' ' 2026-03-09 19:10:03.748619 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.748623 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.748627 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.748632 | controller | - ' exit 1' 2026-03-09 19:10:03.748636 | controller | - ' fi' 2026-03-09 19:10:03.748640 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.748644 | controller | - ' ' 2026-03-09 19:10:03.748649 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.748653 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.748657 | controller | | base64 -d)' 2026-03-09 19:10:03.748662 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.748666 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.748670 | controller | - ' exit 1' 2026-03-09 19:10:03.748674 | controller | - ' fi' 2026-03-09 19:10:03.748679 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.748683 | controller | - ' ' 2026-03-09 19:10:03.748687 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.748699 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.748708 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.748715 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.748720 | controller | --timeout=300s' 2026-03-09 19:10:03.748724 | controller | - ' ' 2026-03-09 19:10:03.748728 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.748733 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.748737 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.748741 | controller | - ' ' 2026-03-09 19:10:03.748746 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748750 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748754 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.748759 | controller | - ' ' 2026-03-09 19:10:03.748763 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.748767 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.748771 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.748776 | controller | available"' 2026-03-09 19:10:03.748780 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.748784 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.748789 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.748793 | controller | - ' exit 1' 2026-03-09 19:10:03.748797 | controller | - ' fi' 2026-03-09 19:10:03.748801 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.748806 | controller | - ' ' 2026-03-09 19:10:03.748810 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.748814 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.748819 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748823 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.748827 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748832 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.748836 | controller | -- \' 2026-03-09 19:10:03.748840 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.748844 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.748849 | controller | - ' logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.748853 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.748857 | controller | - ' logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.748862 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.748866 | controller | found' 2026-03-09 19:10:03.748870 | controller | - ' logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.748874 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.748879 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.748883 | controller | - ' ' 2026-03-09 19:10:03.748887 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.748893 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.748898 | controller | - ' ' 2026-03-09 19:10:03.748902 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.748906 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.748911 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.748915 | controller | - ' exit 1' 2026-03-09 19:10:03.748919 | controller | - ' fi' 2026-03-09 19:10:03.748923 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.748928 | controller | - ' ' 2026-03-09 19:10:03.748932 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.748936 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.748940 | controller | | base64 -d)' 2026-03-09 19:10:03.748945 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.748949 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.748953 | controller | - ' exit 1' 2026-03-09 19:10:03.748958 | controller | - ' fi' 2026-03-09 19:10:03.748962 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.748966 | controller | - ' ' 2026-03-09 19:10:03.748970 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.748975 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.748986 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.748990 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.748994 | controller | --timeout=300s' 2026-03-09 19:10:03.748999 | controller | - ' ' 2026-03-09 19:10:03.749003 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.749009 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.749017 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.749023 | controller | - ' ' 2026-03-09 19:10:03.749029 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749035 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749041 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749047 | controller | - ' ' 2026-03-09 19:10:03.749053 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.749057 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.749061 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.749065 | controller | available"' 2026-03-09 19:10:03.749070 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.749074 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.749078 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.749083 | controller | - ' exit 1' 2026-03-09 19:10:03.749087 | controller | - ' fi' 2026-03-09 19:10:03.749091 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.749095 | controller | - ' ' 2026-03-09 19:10:03.749100 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.749104 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.749111 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749115 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.749119 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749124 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.749128 | controller | -- \' 2026-03-09 19:10:03.749132 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749137 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.749143 | controller | - ' logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.749163 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.749167 | controller | - ' logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.749171 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.749176 | controller | found' 2026-03-09 19:10:03.749180 | controller | - ' logger.go:42: 18:55:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.749184 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.749188 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.749193 | controller | - ' ' 2026-03-09 19:10:03.749199 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.749203 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.749208 | controller | - ' ' 2026-03-09 19:10:03.749212 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.749216 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.749221 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.749225 | controller | - ' exit 1' 2026-03-09 19:10:03.749229 | controller | - ' fi' 2026-03-09 19:10:03.749233 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.749238 | controller | - ' ' 2026-03-09 19:10:03.749242 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.749246 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.749251 | controller | | base64 -d)' 2026-03-09 19:10:03.749255 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.749259 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.749263 | controller | - ' exit 1' 2026-03-09 19:10:03.749268 | controller | - ' fi' 2026-03-09 19:10:03.749272 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.749276 | controller | - ' ' 2026-03-09 19:10:03.749280 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.749285 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.749298 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.749304 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.749309 | controller | --timeout=300s' 2026-03-09 19:10:03.749315 | controller | - ' ' 2026-03-09 19:10:03.749319 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.749323 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.749328 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.749334 | controller | - ' ' 2026-03-09 19:10:03.749338 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749342 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749347 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749351 | controller | - ' ' 2026-03-09 19:10:03.749355 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.749360 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.749364 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.749368 | controller | available"' 2026-03-09 19:10:03.749373 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.749377 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.749381 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.749385 | controller | - ' exit 1' 2026-03-09 19:10:03.749390 | controller | - ' fi' 2026-03-09 19:10:03.749394 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.749398 | controller | - ' ' 2026-03-09 19:10:03.749403 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.749407 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.749411 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.749420 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749424 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.749429 | controller | -- \' 2026-03-09 19:10:03.749435 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749441 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.749447 | controller | - ' logger.go:42: 18:55:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.749453 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.749459 | controller | - ' logger.go:42: 18:55:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.749465 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.749470 | controller | found' 2026-03-09 19:10:03.749476 | controller | - ' logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.749480 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.749484 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.749489 | controller | - ' ' 2026-03-09 19:10:03.749493 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.749497 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.749501 | controller | - ' ' 2026-03-09 19:10:03.749506 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.749510 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.749514 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.749518 | controller | - ' exit 1' 2026-03-09 19:10:03.749523 | controller | - ' fi' 2026-03-09 19:10:03.749527 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.749533 | controller | - ' ' 2026-03-09 19:10:03.749540 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.749544 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.749548 | controller | | base64 -d)' 2026-03-09 19:10:03.749553 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.749557 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.749561 | controller | - ' exit 1' 2026-03-09 19:10:03.749566 | controller | - ' fi' 2026-03-09 19:10:03.749570 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.749574 | controller | - ' ' 2026-03-09 19:10:03.749578 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.749583 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.749587 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.749599 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.749604 | controller | --timeout=300s' 2026-03-09 19:10:03.749608 | controller | - ' ' 2026-03-09 19:10:03.749612 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.749617 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.749621 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.749625 | controller | - ' ' 2026-03-09 19:10:03.749629 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749634 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749638 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749642 | controller | - ' ' 2026-03-09 19:10:03.749646 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.749651 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.749655 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.749659 | controller | available"' 2026-03-09 19:10:03.749663 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.749668 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.749672 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.749676 | controller | - ' exit 1' 2026-03-09 19:10:03.749711 | controller | - ' fi' 2026-03-09 19:10:03.749716 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.749720 | controller | - ' ' 2026-03-09 19:10:03.749724 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.749728 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.749733 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749737 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.749741 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749745 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.749750 | controller | -- \' 2026-03-09 19:10:03.749754 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.749760 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.749765 | controller | - ' logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.749769 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.749773 | controller | - ' logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.749777 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.749782 | controller | found' 2026-03-09 19:10:03.749786 | controller | - ' logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.749790 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.749794 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.749799 | controller | - ' ' 2026-03-09 19:10:03.749803 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.749807 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.749812 | controller | - ' ' 2026-03-09 19:10:03.749816 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.749820 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.749825 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.749829 | controller | - ' exit 1' 2026-03-09 19:10:03.749833 | controller | - ' fi' 2026-03-09 19:10:03.749837 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.749842 | controller | - ' ' 2026-03-09 19:10:03.749846 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.749850 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.749854 | controller | | base64 -d)' 2026-03-09 19:10:03.749859 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.749863 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.749867 | controller | - ' exit 1' 2026-03-09 19:10:03.749872 | controller | - ' fi' 2026-03-09 19:10:03.749876 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.749880 | controller | - ' ' 2026-03-09 19:10:03.749884 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.749889 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.749893 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.749907 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.749912 | controller | --timeout=300s' 2026-03-09 19:10:03.749916 | controller | - ' ' 2026-03-09 19:10:03.749920 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.749924 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.749929 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.749933 | controller | - ' ' 2026-03-09 19:10:03.749937 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749942 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749946 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.749950 | controller | - ' ' 2026-03-09 19:10:03.749956 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.749964 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.749968 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.749972 | controller | available"' 2026-03-09 19:10:03.749976 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.749981 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.749985 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.749989 | controller | - ' exit 1' 2026-03-09 19:10:03.749993 | controller | - ' fi' 2026-03-09 19:10:03.749998 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.750002 | controller | - ' ' 2026-03-09 19:10:03.750007 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.750013 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.750019 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750025 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.750036 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750042 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.750046 | controller | -- \' 2026-03-09 19:10:03.750050 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750054 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.750059 | controller | - ' logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.750063 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.750067 | controller | - ' logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.750071 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.750075 | controller | found' 2026-03-09 19:10:03.750080 | controller | - ' logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.750084 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.750088 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.750093 | controller | - ' ' 2026-03-09 19:10:03.750097 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.750101 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.750106 | controller | - ' ' 2026-03-09 19:10:03.750111 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.750116 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.750121 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.750125 | controller | - ' exit 1' 2026-03-09 19:10:03.750130 | controller | - ' fi' 2026-03-09 19:10:03.750136 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.750140 | controller | - ' ' 2026-03-09 19:10:03.750157 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.750164 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.750168 | controller | | base64 -d)' 2026-03-09 19:10:03.750172 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.750177 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.750181 | controller | - ' exit 1' 2026-03-09 19:10:03.750187 | controller | - ' fi' 2026-03-09 19:10:03.750192 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.750196 | controller | - ' ' 2026-03-09 19:10:03.750200 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.750204 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.750209 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.750213 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.750217 | controller | --timeout=300s' 2026-03-09 19:10:03.750221 | controller | - ' ' 2026-03-09 19:10:03.750234 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.750239 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.750243 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.750247 | controller | - ' ' 2026-03-09 19:10:03.750252 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750259 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750264 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750268 | controller | - ' ' 2026-03-09 19:10:03.750274 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.750278 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.750282 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.750287 | controller | available"' 2026-03-09 19:10:03.750291 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.750295 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.750299 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.750304 | controller | - ' exit 1' 2026-03-09 19:10:03.750308 | controller | - ' fi' 2026-03-09 19:10:03.750312 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.750316 | controller | - ' ' 2026-03-09 19:10:03.750321 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.750325 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.750329 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750333 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.750338 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750342 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.750346 | controller | -- \' 2026-03-09 19:10:03.750350 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750354 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.750359 | controller | - ' logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.750363 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.750367 | controller | - ' logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.750371 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.750378 | controller | found' 2026-03-09 19:10:03.750382 | controller | - ' logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.750387 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.750391 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.750395 | controller | - ' ' 2026-03-09 19:10:03.750399 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.750404 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.750408 | controller | - ' ' 2026-03-09 19:10:03.750412 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.750416 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.750421 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.750425 | controller | - ' exit 1' 2026-03-09 19:10:03.750429 | controller | - ' fi' 2026-03-09 19:10:03.750434 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.750438 | controller | - ' ' 2026-03-09 19:10:03.750442 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.750446 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.750451 | controller | | base64 -d)' 2026-03-09 19:10:03.750455 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.750459 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.750463 | controller | - ' exit 1' 2026-03-09 19:10:03.750468 | controller | - ' fi' 2026-03-09 19:10:03.750472 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.750476 | controller | - ' ' 2026-03-09 19:10:03.750481 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.750485 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.750489 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.750493 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.750498 | controller | --timeout=300s' 2026-03-09 19:10:03.750502 | controller | - ' ' 2026-03-09 19:10:03.750513 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.750518 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.750522 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.750526 | controller | - ' ' 2026-03-09 19:10:03.750532 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750536 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750541 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750545 | controller | - ' ' 2026-03-09 19:10:03.750550 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.750554 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.750560 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.750565 | controller | available"' 2026-03-09 19:10:03.750569 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.750574 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.750578 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.750584 | controller | - ' exit 1' 2026-03-09 19:10:03.750589 | controller | - ' fi' 2026-03-09 19:10:03.750593 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.750597 | controller | - ' ' 2026-03-09 19:10:03.750602 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.750606 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.750610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750614 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.750619 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750623 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.750627 | controller | -- \' 2026-03-09 19:10:03.750631 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750636 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.750640 | controller | - ' logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.750644 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.750648 | controller | - ' logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.750653 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.750657 | controller | found' 2026-03-09 19:10:03.750661 | controller | - ' logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.750665 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.750670 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.750674 | controller | - ' ' 2026-03-09 19:10:03.750678 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.750682 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.750687 | controller | - ' ' 2026-03-09 19:10:03.750691 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.750695 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.750700 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.750704 | controller | - ' exit 1' 2026-03-09 19:10:03.750708 | controller | - ' fi' 2026-03-09 19:10:03.750712 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.750717 | controller | - ' ' 2026-03-09 19:10:03.750721 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.750725 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.750729 | controller | | base64 -d)' 2026-03-09 19:10:03.750734 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.750738 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.750742 | controller | - ' exit 1' 2026-03-09 19:10:03.750747 | controller | - ' fi' 2026-03-09 19:10:03.750751 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.750755 | controller | - ' ' 2026-03-09 19:10:03.750759 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.750764 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.750768 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.750772 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.750778 | controller | --timeout=300s' 2026-03-09 19:10:03.750783 | controller | - ' ' 2026-03-09 19:10:03.750787 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.750799 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.750803 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.750808 | controller | - ' ' 2026-03-09 19:10:03.750812 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750816 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750820 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.750825 | controller | - ' ' 2026-03-09 19:10:03.750829 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.750833 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.750838 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.750842 | controller | available"' 2026-03-09 19:10:03.750846 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.750852 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.750857 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.750861 | controller | - ' exit 1' 2026-03-09 19:10:03.750865 | controller | - ' fi' 2026-03-09 19:10:03.750870 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.750874 | controller | - ' ' 2026-03-09 19:10:03.750878 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.750882 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.750887 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750891 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.750895 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750899 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.750904 | controller | -- \' 2026-03-09 19:10:03.750908 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.750912 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.750916 | controller | - ' logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.750921 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.750927 | controller | - ' logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.750931 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.750935 | controller | found' 2026-03-09 19:10:03.750940 | controller | - ' logger.go:42: 18:55:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.750944 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.750949 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.750953 | controller | - ' ' 2026-03-09 19:10:03.750957 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.750962 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.750968 | controller | - ' ' 2026-03-09 19:10:03.750972 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.750977 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.750981 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.750985 | controller | - ' exit 1' 2026-03-09 19:10:03.750990 | controller | - ' fi' 2026-03-09 19:10:03.750994 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.750998 | controller | - ' ' 2026-03-09 19:10:03.751002 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.751007 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.751013 | controller | | base64 -d)' 2026-03-09 19:10:03.751020 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.751025 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.751031 | controller | - ' exit 1' 2026-03-09 19:10:03.751037 | controller | - ' fi' 2026-03-09 19:10:03.751042 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.751047 | controller | - ' ' 2026-03-09 19:10:03.751051 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.751055 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.751060 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.751064 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.751068 | controller | --timeout=300s' 2026-03-09 19:10:03.751072 | controller | - ' ' 2026-03-09 19:10:03.751077 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.751089 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.751094 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.751098 | controller | - ' ' 2026-03-09 19:10:03.751102 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751109 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751113 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751118 | controller | - ' ' 2026-03-09 19:10:03.751122 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.751126 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.751131 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.751135 | controller | available"' 2026-03-09 19:10:03.751139 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.751157 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.751164 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.751168 | controller | - ' exit 1' 2026-03-09 19:10:03.751173 | controller | - ' fi' 2026-03-09 19:10:03.751177 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.751181 | controller | - ' ' 2026-03-09 19:10:03.751186 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.751190 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.751194 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751201 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.751205 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751209 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.751214 | controller | -- \' 2026-03-09 19:10:03.751218 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751222 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.751226 | controller | - ' logger.go:42: 18:55:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.751231 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.751235 | controller | - ' logger.go:42: 18:55:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.751240 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.751246 | controller | found' 2026-03-09 19:10:03.751252 | controller | - ' logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.751258 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.751264 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.751269 | controller | - ' ' 2026-03-09 19:10:03.751275 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.751281 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.751287 | controller | - ' ' 2026-03-09 19:10:03.751293 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.751298 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.751303 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.751307 | controller | - ' exit 1' 2026-03-09 19:10:03.751311 | controller | - ' fi' 2026-03-09 19:10:03.751315 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.751320 | controller | - ' ' 2026-03-09 19:10:03.751324 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.751328 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.751333 | controller | | base64 -d)' 2026-03-09 19:10:03.751337 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.751341 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.751345 | controller | - ' exit 1' 2026-03-09 19:10:03.751350 | controller | - ' fi' 2026-03-09 19:10:03.751358 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.751362 | controller | - ' ' 2026-03-09 19:10:03.751367 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.751371 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.751375 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.751379 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.751384 | controller | --timeout=300s' 2026-03-09 19:10:03.751388 | controller | - ' ' 2026-03-09 19:10:03.751392 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.751405 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.751410 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.751414 | controller | - ' ' 2026-03-09 19:10:03.751421 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751425 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751429 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751434 | controller | - ' ' 2026-03-09 19:10:03.751438 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.751442 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.751446 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.751451 | controller | available"' 2026-03-09 19:10:03.751455 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.751459 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.751463 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.751468 | controller | - ' exit 1' 2026-03-09 19:10:03.751472 | controller | - ' fi' 2026-03-09 19:10:03.751476 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.751481 | controller | - ' ' 2026-03-09 19:10:03.751485 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.751489 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.751493 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751497 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.751502 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751506 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.751510 | controller | -- \' 2026-03-09 19:10:03.751514 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751519 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.751524 | controller | - ' logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.751529 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.751535 | controller | - ' logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.751541 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.751546 | controller | found' 2026-03-09 19:10:03.751552 | controller | - ' logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.751557 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.751563 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.751569 | controller | - ' ' 2026-03-09 19:10:03.751575 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.751580 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.751584 | controller | - ' ' 2026-03-09 19:10:03.751589 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.751593 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.751597 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.751601 | controller | - ' exit 1' 2026-03-09 19:10:03.751606 | controller | - ' fi' 2026-03-09 19:10:03.751610 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.751614 | controller | - ' ' 2026-03-09 19:10:03.751621 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.751626 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.751630 | controller | | base64 -d)' 2026-03-09 19:10:03.751634 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.751638 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.751643 | controller | - ' exit 1' 2026-03-09 19:10:03.751647 | controller | - ' fi' 2026-03-09 19:10:03.751651 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.751655 | controller | - ' ' 2026-03-09 19:10:03.751660 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.751664 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.751668 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.751673 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.751695 | controller | --timeout=300s' 2026-03-09 19:10:03.751699 | controller | - ' ' 2026-03-09 19:10:03.751704 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.751708 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.751720 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.751725 | controller | - ' ' 2026-03-09 19:10:03.751729 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751733 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751738 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.751742 | controller | - ' ' 2026-03-09 19:10:03.751746 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.751751 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.751755 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.751759 | controller | available"' 2026-03-09 19:10:03.751764 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.751768 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.751772 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.751776 | controller | - ' exit 1' 2026-03-09 19:10:03.751781 | controller | - ' fi' 2026-03-09 19:10:03.751785 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.751789 | controller | - ' ' 2026-03-09 19:10:03.751796 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.751800 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.751805 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751809 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.751813 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751818 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.751822 | controller | -- \' 2026-03-09 19:10:03.751826 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.751833 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.751837 | controller | - ' logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.751842 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.751846 | controller | - ' logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.751850 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.751854 | controller | found' 2026-03-09 19:10:03.751859 | controller | - ' logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.751863 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.751867 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.751872 | controller | - ' ' 2026-03-09 19:10:03.751878 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.751884 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.751889 | controller | - ' ' 2026-03-09 19:10:03.751895 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.751901 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.751906 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.751912 | controller | - ' exit 1' 2026-03-09 19:10:03.751918 | controller | - ' fi' 2026-03-09 19:10:03.751923 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.751927 | controller | - ' ' 2026-03-09 19:10:03.751932 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.751936 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.751940 | controller | | base64 -d)' 2026-03-09 19:10:03.751945 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.751951 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.751955 | controller | - ' exit 1' 2026-03-09 19:10:03.751960 | controller | - ' fi' 2026-03-09 19:10:03.751964 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.751968 | controller | - ' ' 2026-03-09 19:10:03.751972 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.751977 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.751981 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.751985 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.751989 | controller | --timeout=300s' 2026-03-09 19:10:03.751994 | controller | - ' ' 2026-03-09 19:10:03.751998 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.752002 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.752018 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.752025 | controller | - ' ' 2026-03-09 19:10:03.752031 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752037 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752044 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752048 | controller | - ' ' 2026-03-09 19:10:03.752052 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.752059 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.752063 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.752068 | controller | available"' 2026-03-09 19:10:03.752072 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.752076 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.752081 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.752085 | controller | - ' exit 1' 2026-03-09 19:10:03.752089 | controller | - ' fi' 2026-03-09 19:10:03.752093 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.752098 | controller | - ' ' 2026-03-09 19:10:03.752102 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.752106 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.752110 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752115 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.752119 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752123 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.752130 | controller | -- \' 2026-03-09 19:10:03.752135 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752139 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.752156 | controller | - ' logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.752163 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.752168 | controller | - ' logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.752172 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.752176 | controller | found' 2026-03-09 19:10:03.752181 | controller | - ' logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.752185 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.752189 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.752193 | controller | - ' ' 2026-03-09 19:10:03.752198 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.752202 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.752207 | controller | - ' ' 2026-03-09 19:10:03.752212 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.752218 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.752224 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.752229 | controller | - ' exit 1' 2026-03-09 19:10:03.752235 | controller | - ' fi' 2026-03-09 19:10:03.752241 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.752247 | controller | - ' ' 2026-03-09 19:10:03.752255 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.752261 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.752267 | controller | | base64 -d)' 2026-03-09 19:10:03.752273 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.752278 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.752284 | controller | - ' exit 1' 2026-03-09 19:10:03.752290 | controller | - ' fi' 2026-03-09 19:10:03.752298 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.752304 | controller | - ' ' 2026-03-09 19:10:03.752309 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.752315 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.752321 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.752326 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.752332 | controller | --timeout=300s' 2026-03-09 19:10:03.752338 | controller | - ' ' 2026-03-09 19:10:03.752343 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.752349 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.752355 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.752360 | controller | - ' ' 2026-03-09 19:10:03.752378 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752385 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752391 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752397 | controller | - ' ' 2026-03-09 19:10:03.752402 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.752407 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.752411 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.752416 | controller | available"' 2026-03-09 19:10:03.752420 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.752424 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.752428 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.752433 | controller | - ' exit 1' 2026-03-09 19:10:03.752437 | controller | - ' fi' 2026-03-09 19:10:03.752441 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.752446 | controller | - ' ' 2026-03-09 19:10:03.752450 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.752454 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.752458 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752466 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.752470 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752474 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.752479 | controller | -- \' 2026-03-09 19:10:03.752483 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752487 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.752491 | controller | - ' logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.752495 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.752500 | controller | - ' logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.752504 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.752511 | controller | found' 2026-03-09 19:10:03.752515 | controller | - ' logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.752519 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.752524 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.752528 | controller | - ' ' 2026-03-09 19:10:03.752532 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.752536 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.752541 | controller | - ' ' 2026-03-09 19:10:03.752545 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.752549 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.752553 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.752558 | controller | - ' exit 1' 2026-03-09 19:10:03.752564 | controller | - ' fi' 2026-03-09 19:10:03.752568 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.752572 | controller | - ' ' 2026-03-09 19:10:03.752577 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.752581 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.752585 | controller | | base64 -d)' 2026-03-09 19:10:03.752589 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.752594 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.752598 | controller | - ' exit 1' 2026-03-09 19:10:03.752602 | controller | - ' fi' 2026-03-09 19:10:03.752606 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.752611 | controller | - ' ' 2026-03-09 19:10:03.752615 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.752619 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.752623 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.752628 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.752632 | controller | --timeout=300s' 2026-03-09 19:10:03.752636 | controller | - ' ' 2026-03-09 19:10:03.752640 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.752645 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.752649 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.752653 | controller | - ' ' 2026-03-09 19:10:03.752665 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752669 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752674 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752678 | controller | - ' ' 2026-03-09 19:10:03.752682 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.752687 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.752691 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.752695 | controller | available"' 2026-03-09 19:10:03.752699 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.752704 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.752708 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.752712 | controller | - ' exit 1' 2026-03-09 19:10:03.752718 | controller | - ' fi' 2026-03-09 19:10:03.752722 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.752727 | controller | - ' ' 2026-03-09 19:10:03.752731 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.752735 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.752741 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752745 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.752749 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752754 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.752758 | controller | -- \' 2026-03-09 19:10:03.752762 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.752767 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.752771 | controller | - ' logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.752775 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.752779 | controller | - ' logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.752784 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.752788 | controller | found' 2026-03-09 19:10:03.752792 | controller | - ' logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.752797 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.752801 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.752805 | controller | - ' ' 2026-03-09 19:10:03.752809 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.752814 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.752818 | controller | - ' ' 2026-03-09 19:10:03.752822 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.752826 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.752831 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.752835 | controller | - ' exit 1' 2026-03-09 19:10:03.752839 | controller | - ' fi' 2026-03-09 19:10:03.752843 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.752848 | controller | - ' ' 2026-03-09 19:10:03.752852 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.752856 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.752860 | controller | | base64 -d)' 2026-03-09 19:10:03.752865 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.752869 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.752873 | controller | - ' exit 1' 2026-03-09 19:10:03.752878 | controller | - ' fi' 2026-03-09 19:10:03.752882 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.752886 | controller | - ' ' 2026-03-09 19:10:03.752890 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.752895 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.752899 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.752903 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.752907 | controller | --timeout=300s' 2026-03-09 19:10:03.752915 | controller | - ' ' 2026-03-09 19:10:03.752919 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.752923 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.752927 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.752932 | controller | - ' ' 2026-03-09 19:10:03.752936 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752947 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752951 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.752956 | controller | - ' ' 2026-03-09 19:10:03.752960 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.752964 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.752969 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.752973 | controller | available"' 2026-03-09 19:10:03.752977 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.752981 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.752985 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.752990 | controller | - ' exit 1' 2026-03-09 19:10:03.752994 | controller | - ' fi' 2026-03-09 19:10:03.752998 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.753002 | controller | - ' ' 2026-03-09 19:10:03.753008 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.753014 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.753020 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753026 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.753032 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753038 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.753044 | controller | -- \' 2026-03-09 19:10:03.753050 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753059 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.753066 | controller | - ' logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.753072 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.753076 | controller | - ' logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.753080 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.753085 | controller | found' 2026-03-09 19:10:03.753089 | controller | - ' logger.go:42: 18:56:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.753093 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.753097 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.753102 | controller | - ' ' 2026-03-09 19:10:03.753106 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.753110 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.753114 | controller | - ' ' 2026-03-09 19:10:03.753121 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.753125 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.753130 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.753134 | controller | - ' exit 1' 2026-03-09 19:10:03.753138 | controller | - ' fi' 2026-03-09 19:10:03.753142 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.753254 | controller | - ' ' 2026-03-09 19:10:03.753259 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.753263 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.753268 | controller | | base64 -d)' 2026-03-09 19:10:03.753272 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.753276 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.753281 | controller | - ' exit 1' 2026-03-09 19:10:03.753285 | controller | - ' fi' 2026-03-09 19:10:03.753289 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.753293 | controller | - ' ' 2026-03-09 19:10:03.753298 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.753302 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.753306 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.753310 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.753315 | controller | --timeout=300s' 2026-03-09 19:10:03.753319 | controller | - ' ' 2026-03-09 19:10:03.753323 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.753327 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.753332 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.753336 | controller | - ' ' 2026-03-09 19:10:03.753340 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753354 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753361 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753366 | controller | - ' ' 2026-03-09 19:10:03.753370 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.753374 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.753379 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.753383 | controller | available"' 2026-03-09 19:10:03.753387 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.753391 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.753396 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.753400 | controller | - ' exit 1' 2026-03-09 19:10:03.753404 | controller | - ' fi' 2026-03-09 19:10:03.753409 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.753413 | controller | - ' ' 2026-03-09 19:10:03.753417 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.753421 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.753426 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753432 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.753436 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753440 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.753445 | controller | -- \' 2026-03-09 19:10:03.753449 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753453 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.753459 | controller | - ' logger.go:42: 18:56:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.753463 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.753468 | controller | - ' logger.go:42: 18:56:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.753472 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.753476 | controller | found' 2026-03-09 19:10:03.753481 | controller | - ' logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.753485 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.753489 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.753493 | controller | - ' ' 2026-03-09 19:10:03.753498 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.753502 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.753506 | controller | - ' ' 2026-03-09 19:10:03.753510 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.753515 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.753519 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.753523 | controller | - ' exit 1' 2026-03-09 19:10:03.753528 | controller | - ' fi' 2026-03-09 19:10:03.753532 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.753536 | controller | - ' ' 2026-03-09 19:10:03.753540 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.753545 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.753549 | controller | | base64 -d)' 2026-03-09 19:10:03.753553 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.753557 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.753562 | controller | - ' exit 1' 2026-03-09 19:10:03.753566 | controller | - ' fi' 2026-03-09 19:10:03.753570 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.753575 | controller | - ' ' 2026-03-09 19:10:03.753579 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.753583 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.753587 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.753592 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.753596 | controller | --timeout=300s' 2026-03-09 19:10:03.753602 | controller | - ' ' 2026-03-09 19:10:03.753606 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.753611 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.753615 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.753619 | controller | - ' ' 2026-03-09 19:10:03.753623 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753630 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753641 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753645 | controller | - ' ' 2026-03-09 19:10:03.753650 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.753654 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.753658 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.753662 | controller | available"' 2026-03-09 19:10:03.753667 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.753671 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.753676 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.753680 | controller | - ' exit 1' 2026-03-09 19:10:03.753687 | controller | - ' fi' 2026-03-09 19:10:03.753692 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.753696 | controller | - ' ' 2026-03-09 19:10:03.753700 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.753704 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.753709 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753713 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.753717 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753721 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.753726 | controller | -- \' 2026-03-09 19:10:03.753730 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753734 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.753738 | controller | - ' logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.753743 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.753747 | controller | - ' logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.753751 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.753755 | controller | found' 2026-03-09 19:10:03.753760 | controller | - ' logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.753764 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.753770 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.753774 | controller | - ' ' 2026-03-09 19:10:03.753778 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.753782 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.753787 | controller | - ' ' 2026-03-09 19:10:03.753791 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.753795 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.753800 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.753804 | controller | - ' exit 1' 2026-03-09 19:10:03.753808 | controller | - ' fi' 2026-03-09 19:10:03.753812 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.753817 | controller | - ' ' 2026-03-09 19:10:03.753821 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.753835 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.753840 | controller | | base64 -d)' 2026-03-09 19:10:03.753844 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.753848 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.753853 | controller | - ' exit 1' 2026-03-09 19:10:03.753857 | controller | - ' fi' 2026-03-09 19:10:03.753861 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.753865 | controller | - ' ' 2026-03-09 19:10:03.753870 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.753874 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.753878 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.753883 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.753887 | controller | --timeout=300s' 2026-03-09 19:10:03.753891 | controller | - ' ' 2026-03-09 19:10:03.753895 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.753900 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.753904 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.753908 | controller | - ' ' 2026-03-09 19:10:03.753912 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753917 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753928 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.753932 | controller | - ' ' 2026-03-09 19:10:03.753936 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.753941 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.753945 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.753949 | controller | available"' 2026-03-09 19:10:03.753954 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.753958 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.753962 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.753966 | controller | - ' exit 1' 2026-03-09 19:10:03.753971 | controller | - ' fi' 2026-03-09 19:10:03.753975 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.753979 | controller | - ' ' 2026-03-09 19:10:03.753983 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.753988 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.753992 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.753996 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.754000 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754005 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.754011 | controller | -- \' 2026-03-09 19:10:03.754017 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754026 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.754032 | controller | - ' logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.754038 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.754044 | controller | - ' logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.754050 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.754055 | controller | found' 2026-03-09 19:10:03.754061 | controller | - ' logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.754066 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.754072 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.754077 | controller | - ' ' 2026-03-09 19:10:03.754082 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.754088 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.754094 | controller | - ' ' 2026-03-09 19:10:03.754098 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.754102 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.754106 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.754111 | controller | - ' exit 1' 2026-03-09 19:10:03.754115 | controller | - ' fi' 2026-03-09 19:10:03.754119 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.754124 | controller | - ' ' 2026-03-09 19:10:03.754128 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.754132 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.754136 | controller | | base64 -d)' 2026-03-09 19:10:03.754141 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.754156 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.754163 | controller | - ' exit 1' 2026-03-09 19:10:03.754167 | controller | - ' fi' 2026-03-09 19:10:03.754172 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.754176 | controller | - ' ' 2026-03-09 19:10:03.754180 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.754184 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.754189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.754193 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.754197 | controller | --timeout=300s' 2026-03-09 19:10:03.754201 | controller | - ' ' 2026-03-09 19:10:03.754205 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.754210 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.754214 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.754218 | controller | - ' ' 2026-03-09 19:10:03.754222 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754227 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754239 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754244 | controller | - ' ' 2026-03-09 19:10:03.754248 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.754252 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.754260 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.754264 | controller | available"' 2026-03-09 19:10:03.754269 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.754273 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.754277 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.754281 | controller | - ' exit 1' 2026-03-09 19:10:03.754286 | controller | - ' fi' 2026-03-09 19:10:03.754290 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.754294 | controller | - ' ' 2026-03-09 19:10:03.754299 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.754303 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.754307 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754311 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.754316 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754320 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.754324 | controller | -- \' 2026-03-09 19:10:03.754328 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754333 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.754339 | controller | - ' logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.754344 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.754348 | controller | - ' logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.754352 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.754360 | controller | found' 2026-03-09 19:10:03.754364 | controller | - ' logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.754368 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.754373 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.754377 | controller | - ' ' 2026-03-09 19:10:03.754381 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.754385 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.754390 | controller | - ' ' 2026-03-09 19:10:03.754394 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.754398 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.754402 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.754407 | controller | - ' exit 1' 2026-03-09 19:10:03.754411 | controller | - ' fi' 2026-03-09 19:10:03.754415 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.754420 | controller | - ' ' 2026-03-09 19:10:03.754424 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.754428 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.754432 | controller | | base64 -d)' 2026-03-09 19:10:03.754437 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.754441 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.754445 | controller | - ' exit 1' 2026-03-09 19:10:03.754449 | controller | - ' fi' 2026-03-09 19:10:03.754453 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.754459 | controller | - ' ' 2026-03-09 19:10:03.754464 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.754468 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.754472 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.754476 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.754481 | controller | --timeout=300s' 2026-03-09 19:10:03.754485 | controller | - ' ' 2026-03-09 19:10:03.754489 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.754494 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.754498 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.754502 | controller | - ' ' 2026-03-09 19:10:03.754506 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754511 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754515 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754519 | controller | - ' ' 2026-03-09 19:10:03.754530 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.754535 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.754539 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.754543 | controller | available"' 2026-03-09 19:10:03.754547 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.754552 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.754556 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.754560 | controller | - ' exit 1' 2026-03-09 19:10:03.754564 | controller | - ' fi' 2026-03-09 19:10:03.754569 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.754573 | controller | - ' ' 2026-03-09 19:10:03.754577 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.754581 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.754585 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754590 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.754594 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754598 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.754602 | controller | -- \' 2026-03-09 19:10:03.754607 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754611 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.754615 | controller | - ' logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.754619 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.754624 | controller | - ' logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.754628 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.754632 | controller | found' 2026-03-09 19:10:03.754636 | controller | - ' logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.754643 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.754647 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.754653 | controller | - ' ' 2026-03-09 19:10:03.754657 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.754662 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.754666 | controller | - ' ' 2026-03-09 19:10:03.754670 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.754675 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.754679 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.754684 | controller | - ' exit 1' 2026-03-09 19:10:03.754688 | controller | - ' fi' 2026-03-09 19:10:03.754692 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.754696 | controller | - ' ' 2026-03-09 19:10:03.754701 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.754705 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.754709 | controller | | base64 -d)' 2026-03-09 19:10:03.754713 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.754718 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.754722 | controller | - ' exit 1' 2026-03-09 19:10:03.754726 | controller | - ' fi' 2026-03-09 19:10:03.754731 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.754735 | controller | - ' ' 2026-03-09 19:10:03.754739 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.754743 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.754748 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.754752 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.754756 | controller | --timeout=300s' 2026-03-09 19:10:03.754762 | controller | - ' ' 2026-03-09 19:10:03.754767 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.754771 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.754775 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.754780 | controller | - ' ' 2026-03-09 19:10:03.754786 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754790 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754794 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.754799 | controller | - ' ' 2026-03-09 19:10:03.754810 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.754814 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.754819 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.754823 | controller | available"' 2026-03-09 19:10:03.754827 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.754831 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.754836 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.754840 | controller | - ' exit 1' 2026-03-09 19:10:03.754845 | controller | - ' fi' 2026-03-09 19:10:03.754852 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.754856 | controller | - ' ' 2026-03-09 19:10:03.754861 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.754865 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.754869 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754874 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.754878 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754882 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.754886 | controller | -- \' 2026-03-09 19:10:03.754891 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.754895 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.754899 | controller | - ' logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.754904 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.754908 | controller | - ' logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.754912 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.754917 | controller | found' 2026-03-09 19:10:03.754923 | controller | - ' logger.go:42: 18:56:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.754927 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.754931 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.754936 | controller | - ' ' 2026-03-09 19:10:03.754940 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.754944 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.754948 | controller | - ' ' 2026-03-09 19:10:03.754953 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.754957 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.754961 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.754966 | controller | - ' exit 1' 2026-03-09 19:10:03.754970 | controller | - ' fi' 2026-03-09 19:10:03.754974 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.754978 | controller | - ' ' 2026-03-09 19:10:03.754984 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.754988 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.754993 | controller | | base64 -d)' 2026-03-09 19:10:03.754997 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.755001 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.755006 | controller | - ' exit 1' 2026-03-09 19:10:03.755012 | controller | - ' fi' 2026-03-09 19:10:03.755018 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.755024 | controller | - ' ' 2026-03-09 19:10:03.755030 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.755036 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.755042 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.755046 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.755050 | controller | --timeout=300s' 2026-03-09 19:10:03.755055 | controller | - ' ' 2026-03-09 19:10:03.755061 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.755065 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.755069 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.755074 | controller | - ' ' 2026-03-09 19:10:03.755078 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755082 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755086 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755091 | controller | - ' ' 2026-03-09 19:10:03.755095 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.755107 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.755112 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.755116 | controller | available"' 2026-03-09 19:10:03.755120 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.755125 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.755129 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.755133 | controller | - ' exit 1' 2026-03-09 19:10:03.755138 | controller | - ' fi' 2026-03-09 19:10:03.755142 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.755160 | controller | - ' ' 2026-03-09 19:10:03.755164 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.755168 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.755173 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755177 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.755181 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755185 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.755190 | controller | -- \' 2026-03-09 19:10:03.755194 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755198 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.755202 | controller | - ' logger.go:42: 18:56:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.755207 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.755211 | controller | - ' logger.go:42: 18:56:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.755215 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.755219 | controller | found' 2026-03-09 19:10:03.755224 | controller | - ' logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.755228 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.755232 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.755236 | controller | - ' ' 2026-03-09 19:10:03.755243 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.755248 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.755252 | controller | - ' ' 2026-03-09 19:10:03.755257 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.755263 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.755267 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.755271 | controller | - ' exit 1' 2026-03-09 19:10:03.755276 | controller | - ' fi' 2026-03-09 19:10:03.755280 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.755284 | controller | - ' ' 2026-03-09 19:10:03.755289 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.755293 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.755297 | controller | | base64 -d)' 2026-03-09 19:10:03.755301 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.755306 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.755310 | controller | - ' exit 1' 2026-03-09 19:10:03.755314 | controller | - ' fi' 2026-03-09 19:10:03.755318 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.755323 | controller | - ' ' 2026-03-09 19:10:03.755327 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.755331 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.755335 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.755340 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.755344 | controller | --timeout=300s' 2026-03-09 19:10:03.755348 | controller | - ' ' 2026-03-09 19:10:03.755352 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.755357 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.755361 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.755365 | controller | - ' ' 2026-03-09 19:10:03.755369 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755374 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755378 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755382 | controller | - ' ' 2026-03-09 19:10:03.755387 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.755393 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.755405 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.755409 | controller | available"' 2026-03-09 19:10:03.755413 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.755418 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.755422 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.755426 | controller | - ' exit 1' 2026-03-09 19:10:03.755431 | controller | - ' fi' 2026-03-09 19:10:03.755435 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.755439 | controller | - ' ' 2026-03-09 19:10:03.755443 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.755448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.755452 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.755462 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755466 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.755471 | controller | -- \' 2026-03-09 19:10:03.755475 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755479 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.755484 | controller | - ' logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.755488 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.755492 | controller | - ' logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.755496 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.755501 | controller | found' 2026-03-09 19:10:03.755505 | controller | - ' logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.755509 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.755513 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.755518 | controller | - ' ' 2026-03-09 19:10:03.755524 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.755528 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.755532 | controller | - ' ' 2026-03-09 19:10:03.755537 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.755541 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.755545 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.755549 | controller | - ' exit 1' 2026-03-09 19:10:03.755554 | controller | - ' fi' 2026-03-09 19:10:03.755558 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.755562 | controller | - ' ' 2026-03-09 19:10:03.755567 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.755571 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.755575 | controller | | base64 -d)' 2026-03-09 19:10:03.755579 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.755584 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.755588 | controller | - ' exit 1' 2026-03-09 19:10:03.755592 | controller | - ' fi' 2026-03-09 19:10:03.755596 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.755601 | controller | - ' ' 2026-03-09 19:10:03.755605 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.755609 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.755613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.755618 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.755622 | controller | --timeout=300s' 2026-03-09 19:10:03.755626 | controller | - ' ' 2026-03-09 19:10:03.755630 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.755635 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.755639 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.755643 | controller | - ' ' 2026-03-09 19:10:03.755647 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755654 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755659 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755663 | controller | - ' ' 2026-03-09 19:10:03.755667 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.755685 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.755698 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.755702 | controller | available"' 2026-03-09 19:10:03.755706 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.755711 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.755715 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.755719 | controller | - ' exit 1' 2026-03-09 19:10:03.755755 | controller | - ' fi' 2026-03-09 19:10:03.755760 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.755764 | controller | - ' ' 2026-03-09 19:10:03.755768 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.755779 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.755783 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755788 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.755792 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755796 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.755801 | controller | -- \' 2026-03-09 19:10:03.755805 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.755809 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.755813 | controller | - ' logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.755818 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.755822 | controller | - ' logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.755826 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.755830 | controller | found' 2026-03-09 19:10:03.755835 | controller | - ' logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.755839 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.755843 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.755848 | controller | - ' ' 2026-03-09 19:10:03.755852 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.755856 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.755860 | controller | - ' ' 2026-03-09 19:10:03.755865 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.755869 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.755873 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.755877 | controller | - ' exit 1' 2026-03-09 19:10:03.755882 | controller | - ' fi' 2026-03-09 19:10:03.755886 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.755890 | controller | - ' ' 2026-03-09 19:10:03.755907 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.755912 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.755918 | controller | | base64 -d)' 2026-03-09 19:10:03.755922 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.755926 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.755931 | controller | - ' exit 1' 2026-03-09 19:10:03.755935 | controller | - ' fi' 2026-03-09 19:10:03.755939 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.755944 | controller | - ' ' 2026-03-09 19:10:03.755948 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.755952 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.755956 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.755960 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.755965 | controller | --timeout=300s' 2026-03-09 19:10:03.755969 | controller | - ' ' 2026-03-09 19:10:03.755973 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.755977 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.755982 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.755986 | controller | - ' ' 2026-03-09 19:10:03.755990 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.755995 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756017 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756024 | controller | - ' ' 2026-03-09 19:10:03.756030 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.756036 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.756042 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.756047 | controller | available"' 2026-03-09 19:10:03.756066 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.756072 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.756077 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.756083 | controller | - ' exit 1' 2026-03-09 19:10:03.756087 | controller | - ' fi' 2026-03-09 19:10:03.756091 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.756095 | controller | - ' ' 2026-03-09 19:10:03.756100 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.756104 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.756108 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756112 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.756117 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756121 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.756125 | controller | -- \' 2026-03-09 19:10:03.756129 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756134 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.756141 | controller | - ' logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.756165 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.756172 | controller | - ' logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.756176 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.756181 | controller | found' 2026-03-09 19:10:03.756185 | controller | - ' logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.756189 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.756193 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.756198 | controller | - ' ' 2026-03-09 19:10:03.756202 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.756209 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.756213 | controller | - ' ' 2026-03-09 19:10:03.756218 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.756222 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.756226 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.756230 | controller | - ' exit 1' 2026-03-09 19:10:03.756235 | controller | - ' fi' 2026-03-09 19:10:03.756239 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.756243 | controller | - ' ' 2026-03-09 19:10:03.756247 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.756252 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.756256 | controller | | base64 -d)' 2026-03-09 19:10:03.756260 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.756266 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.756270 | controller | - ' exit 1' 2026-03-09 19:10:03.756275 | controller | - ' fi' 2026-03-09 19:10:03.756279 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.756283 | controller | - ' ' 2026-03-09 19:10:03.756288 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.756292 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.756296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.756300 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.756305 | controller | --timeout=300s' 2026-03-09 19:10:03.756309 | controller | - ' ' 2026-03-09 19:10:03.756313 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.756317 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.756322 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.756326 | controller | - ' ' 2026-03-09 19:10:03.756330 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756334 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756339 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756343 | controller | - ' ' 2026-03-09 19:10:03.756347 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.756351 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.756356 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.756362 | controller | available"' 2026-03-09 19:10:03.756366 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.756383 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.756388 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.756393 | controller | - ' exit 1' 2026-03-09 19:10:03.756397 | controller | - ' fi' 2026-03-09 19:10:03.756401 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.756405 | controller | - ' ' 2026-03-09 19:10:03.756410 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.756414 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.756418 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756422 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.756426 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756431 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.756435 | controller | -- \' 2026-03-09 19:10:03.756439 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756444 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.756448 | controller | - ' logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.756452 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.756457 | controller | - ' logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.756461 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.756465 | controller | found' 2026-03-09 19:10:03.756469 | controller | - ' logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.756474 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.756490 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.756495 | controller | - ' ' 2026-03-09 19:10:03.756499 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.756503 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.756507 | controller | - ' ' 2026-03-09 19:10:03.756512 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.756516 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.756520 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.756524 | controller | - ' exit 1' 2026-03-09 19:10:03.756529 | controller | - ' fi' 2026-03-09 19:10:03.756533 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.756537 | controller | - ' ' 2026-03-09 19:10:03.756541 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.756546 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.756550 | controller | | base64 -d)' 2026-03-09 19:10:03.756554 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.756558 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.756562 | controller | - ' exit 1' 2026-03-09 19:10:03.756567 | controller | - ' fi' 2026-03-09 19:10:03.756571 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.756575 | controller | - ' ' 2026-03-09 19:10:03.756581 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.756586 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.756590 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.756594 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.756598 | controller | --timeout=300s' 2026-03-09 19:10:03.756603 | controller | - ' ' 2026-03-09 19:10:03.756607 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.756611 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.756617 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.756621 | controller | - ' ' 2026-03-09 19:10:03.756625 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756630 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756634 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756638 | controller | - ' ' 2026-03-09 19:10:03.756642 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.756647 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.756651 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.756655 | controller | available"' 2026-03-09 19:10:03.756659 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.756670 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.756675 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.756679 | controller | - ' exit 1' 2026-03-09 19:10:03.756683 | controller | - ' fi' 2026-03-09 19:10:03.756688 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.756692 | controller | - ' ' 2026-03-09 19:10:03.756696 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.756700 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.756705 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756709 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.756713 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756717 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.756722 | controller | -- \' 2026-03-09 19:10:03.756726 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756730 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.756734 | controller | - ' logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.756739 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.756743 | controller | - ' logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.756747 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.756751 | controller | found' 2026-03-09 19:10:03.756756 | controller | - ' logger.go:42: 18:56:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.756761 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.756766 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.756770 | controller | - ' ' 2026-03-09 19:10:03.756774 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.756779 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.756783 | controller | - ' ' 2026-03-09 19:10:03.756789 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.756793 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.756797 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.756802 | controller | - ' exit 1' 2026-03-09 19:10:03.756806 | controller | - ' fi' 2026-03-09 19:10:03.756810 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.756814 | controller | - ' ' 2026-03-09 19:10:03.756819 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.756823 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.756827 | controller | | base64 -d)' 2026-03-09 19:10:03.756831 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.756836 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.756840 | controller | - ' exit 1' 2026-03-09 19:10:03.756844 | controller | - ' fi' 2026-03-09 19:10:03.756848 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.756853 | controller | - ' ' 2026-03-09 19:10:03.756857 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.756861 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.756865 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.756870 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.756874 | controller | --timeout=300s' 2026-03-09 19:10:03.756878 | controller | - ' ' 2026-03-09 19:10:03.756882 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.756887 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.756891 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.756895 | controller | - ' ' 2026-03-09 19:10:03.756899 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756904 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756908 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.756912 | controller | - ' ' 2026-03-09 19:10:03.756916 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.756920 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.756925 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.756929 | controller | available"' 2026-03-09 19:10:03.756933 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.756937 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.756948 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.756953 | controller | - ' exit 1' 2026-03-09 19:10:03.756957 | controller | - ' fi' 2026-03-09 19:10:03.756961 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.756967 | controller | - ' ' 2026-03-09 19:10:03.756973 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.756977 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.756982 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756986 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.756990 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.756995 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.756999 | controller | -- \' 2026-03-09 19:10:03.757003 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757009 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.757015 | controller | - ' logger.go:42: 18:56:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.757021 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.757026 | controller | - ' logger.go:42: 18:56:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.757032 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.757038 | controller | found' 2026-03-09 19:10:03.757044 | controller | - ' logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.757048 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.757053 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.757057 | controller | - ' ' 2026-03-09 19:10:03.757061 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.757066 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.757070 | controller | - ' ' 2026-03-09 19:10:03.757074 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.757078 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.757083 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.757087 | controller | - ' exit 1' 2026-03-09 19:10:03.757091 | controller | - ' fi' 2026-03-09 19:10:03.757095 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.757100 | controller | - ' ' 2026-03-09 19:10:03.757104 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.757108 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.757112 | controller | | base64 -d)' 2026-03-09 19:10:03.757117 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.757121 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.757125 | controller | - ' exit 1' 2026-03-09 19:10:03.757129 | controller | - ' fi' 2026-03-09 19:10:03.757134 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.757138 | controller | - ' ' 2026-03-09 19:10:03.757142 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.757166 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.757171 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.757175 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.757179 | controller | --timeout=300s' 2026-03-09 19:10:03.757184 | controller | - ' ' 2026-03-09 19:10:03.757188 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.757194 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.757199 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.757203 | controller | - ' ' 2026-03-09 19:10:03.757207 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757212 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757216 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757220 | controller | - ' ' 2026-03-09 19:10:03.757224 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.757229 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.757233 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.757237 | controller | available"' 2026-03-09 19:10:03.757241 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.757246 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.757250 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.757262 | controller | - ' exit 1' 2026-03-09 19:10:03.757267 | controller | - ' fi' 2026-03-09 19:10:03.757271 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.757275 | controller | - ' ' 2026-03-09 19:10:03.757280 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.757284 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.757288 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757293 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.757297 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757301 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.757305 | controller | -- \' 2026-03-09 19:10:03.757310 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757314 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.757318 | controller | - ' logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.757322 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.757327 | controller | - ' logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.757331 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.757335 | controller | found' 2026-03-09 19:10:03.757339 | controller | - ' logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.757344 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.757348 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.757352 | controller | - ' ' 2026-03-09 19:10:03.757356 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.757361 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.757365 | controller | - ' ' 2026-03-09 19:10:03.757369 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.757377 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.757381 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.757386 | controller | - ' exit 1' 2026-03-09 19:10:03.757390 | controller | - ' fi' 2026-03-09 19:10:03.757394 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.757401 | controller | - ' ' 2026-03-09 19:10:03.757405 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.757409 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.757413 | controller | | base64 -d)' 2026-03-09 19:10:03.757418 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.757422 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.757426 | controller | - ' exit 1' 2026-03-09 19:10:03.757431 | controller | - ' fi' 2026-03-09 19:10:03.757435 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.757439 | controller | - ' ' 2026-03-09 19:10:03.757443 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.757448 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.757452 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.757456 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.757460 | controller | --timeout=300s' 2026-03-09 19:10:03.757465 | controller | - ' ' 2026-03-09 19:10:03.757469 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.757473 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.757477 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.757482 | controller | - ' ' 2026-03-09 19:10:03.757486 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757490 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757494 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757499 | controller | - ' ' 2026-03-09 19:10:03.757503 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.757507 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.757511 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.757516 | controller | available"' 2026-03-09 19:10:03.757520 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.757524 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.757528 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.757533 | controller | - ' exit 1' 2026-03-09 19:10:03.757537 | controller | - ' fi' 2026-03-09 19:10:03.757548 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.757553 | controller | - ' ' 2026-03-09 19:10:03.757557 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.757561 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.757565 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757570 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.757576 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757582 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.757586 | controller | -- \' 2026-03-09 19:10:03.757592 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757596 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.757602 | controller | - ' logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.757606 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.757611 | controller | - ' logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.757615 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.757619 | controller | found' 2026-03-09 19:10:03.757624 | controller | - ' logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.757628 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.757632 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.757636 | controller | - ' ' 2026-03-09 19:10:03.757641 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.757645 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.757649 | controller | - ' ' 2026-03-09 19:10:03.757653 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.757658 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.757662 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.757666 | controller | - ' exit 1' 2026-03-09 19:10:03.757670 | controller | - ' fi' 2026-03-09 19:10:03.757675 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.757679 | controller | - ' ' 2026-03-09 19:10:03.757685 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.757689 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.757693 | controller | | base64 -d)' 2026-03-09 19:10:03.757698 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.757702 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.757706 | controller | - ' exit 1' 2026-03-09 19:10:03.757712 | controller | - ' fi' 2026-03-09 19:10:03.757716 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.757720 | controller | - ' ' 2026-03-09 19:10:03.757725 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.757729 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.757733 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.757737 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.757742 | controller | --timeout=300s' 2026-03-09 19:10:03.757747 | controller | - ' ' 2026-03-09 19:10:03.757752 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.757756 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.757760 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.757765 | controller | - ' ' 2026-03-09 19:10:03.757769 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757773 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757779 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.757783 | controller | - ' ' 2026-03-09 19:10:03.757788 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.757792 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.757796 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.757800 | controller | available"' 2026-03-09 19:10:03.757805 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.757809 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.757813 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.757817 | controller | - ' exit 1' 2026-03-09 19:10:03.757822 | controller | - ' fi' 2026-03-09 19:10:03.757833 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.757838 | controller | - ' ' 2026-03-09 19:10:03.757842 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.757847 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.757851 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757855 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.757860 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757864 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.757868 | controller | -- \' 2026-03-09 19:10:03.757874 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.757878 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.757882 | controller | - ' logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.757887 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.757891 | controller | - ' logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.757895 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.757899 | controller | found' 2026-03-09 19:10:03.757904 | controller | - ' logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.757908 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.757912 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.757917 | controller | - ' ' 2026-03-09 19:10:03.757921 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.757925 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.757930 | controller | - ' ' 2026-03-09 19:10:03.757934 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.757938 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.757942 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.757947 | controller | - ' exit 1' 2026-03-09 19:10:03.757951 | controller | - ' fi' 2026-03-09 19:10:03.757955 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.757959 | controller | - ' ' 2026-03-09 19:10:03.757964 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.757968 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.757974 | controller | | base64 -d)' 2026-03-09 19:10:03.757978 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.757983 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.757987 | controller | - ' exit 1' 2026-03-09 19:10:03.757991 | controller | - ' fi' 2026-03-09 19:10:03.757995 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.758000 | controller | - ' ' 2026-03-09 19:10:03.758004 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.758010 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.758018 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.758024 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.758031 | controller | --timeout=300s' 2026-03-09 19:10:03.758036 | controller | - ' ' 2026-03-09 19:10:03.758041 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.758045 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.758049 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.758054 | controller | - ' ' 2026-03-09 19:10:03.758060 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758064 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758068 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758073 | controller | - ' ' 2026-03-09 19:10:03.758077 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.758081 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.758086 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.758090 | controller | available"' 2026-03-09 19:10:03.758094 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.758098 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.758102 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.758107 | controller | - ' exit 1' 2026-03-09 19:10:03.758111 | controller | - ' fi' 2026-03-09 19:10:03.758115 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.758119 | controller | - ' ' 2026-03-09 19:10:03.758132 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.758136 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.758140 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758156 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.758163 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758167 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.758172 | controller | -- \' 2026-03-09 19:10:03.758186 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758191 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.758195 | controller | - ' logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.758203 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.758207 | controller | - ' logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.758211 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.758216 | controller | found' 2026-03-09 19:10:03.758220 | controller | - ' logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.758224 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.758228 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.758233 | controller | - ' ' 2026-03-09 19:10:03.758237 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.758241 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.758246 | controller | - ' ' 2026-03-09 19:10:03.758250 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.758254 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.758258 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.758263 | controller | - ' exit 1' 2026-03-09 19:10:03.758267 | controller | - ' fi' 2026-03-09 19:10:03.758271 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.758276 | controller | - ' ' 2026-03-09 19:10:03.758280 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.758284 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.758288 | controller | | base64 -d)' 2026-03-09 19:10:03.758293 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.758297 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.758301 | controller | - ' exit 1' 2026-03-09 19:10:03.758305 | controller | - ' fi' 2026-03-09 19:10:03.758310 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.758314 | controller | - ' ' 2026-03-09 19:10:03.758318 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.758322 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.758327 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.758331 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.758335 | controller | --timeout=300s' 2026-03-09 19:10:03.758339 | controller | - ' ' 2026-03-09 19:10:03.758344 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.758348 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.758352 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.758356 | controller | - ' ' 2026-03-09 19:10:03.758361 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758365 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758369 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758373 | controller | - ' ' 2026-03-09 19:10:03.758378 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.758382 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.758386 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.758392 | controller | available"' 2026-03-09 19:10:03.758396 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.758401 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.758405 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.758409 | controller | - ' exit 1' 2026-03-09 19:10:03.758413 | controller | - ' fi' 2026-03-09 19:10:03.758418 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.758422 | controller | - ' ' 2026-03-09 19:10:03.758434 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.758439 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.758443 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.758452 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.758461 | controller | -- \' 2026-03-09 19:10:03.758465 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758469 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.758473 | controller | - ' logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.758477 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.758484 | controller | - ' logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.758488 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.758493 | controller | found' 2026-03-09 19:10:03.758497 | controller | - ' logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.758501 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.758506 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.758510 | controller | - ' ' 2026-03-09 19:10:03.758514 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.758519 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.758523 | controller | - ' ' 2026-03-09 19:10:03.758527 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.758531 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.758536 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.758540 | controller | - ' exit 1' 2026-03-09 19:10:03.758544 | controller | - ' fi' 2026-03-09 19:10:03.758548 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.758553 | controller | - ' ' 2026-03-09 19:10:03.758557 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.758561 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.758565 | controller | | base64 -d)' 2026-03-09 19:10:03.758570 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.758574 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.758578 | controller | - ' exit 1' 2026-03-09 19:10:03.758582 | controller | - ' fi' 2026-03-09 19:10:03.758587 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.758591 | controller | - ' ' 2026-03-09 19:10:03.758595 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.758603 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.758607 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.758613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.758617 | controller | --timeout=300s' 2026-03-09 19:10:03.758621 | controller | - ' ' 2026-03-09 19:10:03.758625 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.758630 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.758634 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.758638 | controller | - ' ' 2026-03-09 19:10:03.758642 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758648 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758653 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758657 | controller | - ' ' 2026-03-09 19:10:03.758661 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.758665 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.758670 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.758674 | controller | available"' 2026-03-09 19:10:03.758678 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.758682 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.758687 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.758691 | controller | - ' exit 1' 2026-03-09 19:10:03.758695 | controller | - ' fi' 2026-03-09 19:10:03.758699 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.758704 | controller | - ' ' 2026-03-09 19:10:03.758708 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.758719 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.758723 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758727 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.758732 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758736 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.758740 | controller | -- \' 2026-03-09 19:10:03.758744 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.758749 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.758753 | controller | - ' logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.758757 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.758761 | controller | - ' logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.758766 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.758770 | controller | found' 2026-03-09 19:10:03.758774 | controller | - ' logger.go:42: 18:56:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.758778 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.758785 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.758789 | controller | - ' ' 2026-03-09 19:10:03.758793 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.758797 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.758801 | controller | - ' ' 2026-03-09 19:10:03.758806 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.758810 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.758814 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.758819 | controller | - ' exit 1' 2026-03-09 19:10:03.758823 | controller | - ' fi' 2026-03-09 19:10:03.758827 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.758831 | controller | - ' ' 2026-03-09 19:10:03.758836 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.758840 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.758844 | controller | | base64 -d)' 2026-03-09 19:10:03.758848 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.758852 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.758857 | controller | - ' exit 1' 2026-03-09 19:10:03.758861 | controller | - ' fi' 2026-03-09 19:10:03.758865 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.758869 | controller | - ' ' 2026-03-09 19:10:03.758874 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.758878 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.758882 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.758886 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.758891 | controller | --timeout=300s' 2026-03-09 19:10:03.758895 | controller | - ' ' 2026-03-09 19:10:03.758899 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.758903 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.758908 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.758912 | controller | - ' ' 2026-03-09 19:10:03.758918 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758922 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758926 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.758930 | controller | - ' ' 2026-03-09 19:10:03.758935 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.758939 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.758943 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.758947 | controller | available"' 2026-03-09 19:10:03.758952 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.758957 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.758962 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.758966 | controller | - ' exit 1' 2026-03-09 19:10:03.758970 | controller | - ' fi' 2026-03-09 19:10:03.758974 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.758979 | controller | - ' ' 2026-03-09 19:10:03.758984 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.758989 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.759000 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759005 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.759011 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759017 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.759024 | controller | -- \' 2026-03-09 19:10:03.759030 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759036 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.759041 | controller | - ' logger.go:42: 18:56:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.759047 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.759053 | controller | - ' logger.go:42: 18:56:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.759058 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.759064 | controller | found' 2026-03-09 19:10:03.759069 | controller | - ' logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.759075 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.759081 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.759086 | controller | - ' ' 2026-03-09 19:10:03.759090 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.759094 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.759098 | controller | - ' ' 2026-03-09 19:10:03.759103 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.759107 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.759111 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.759116 | controller | - ' exit 1' 2026-03-09 19:10:03.759120 | controller | - ' fi' 2026-03-09 19:10:03.759124 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.759128 | controller | - ' ' 2026-03-09 19:10:03.759133 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.759137 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.759141 | controller | | base64 -d)' 2026-03-09 19:10:03.759163 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.759171 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.759175 | controller | - ' exit 1' 2026-03-09 19:10:03.759182 | controller | - ' fi' 2026-03-09 19:10:03.759187 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.759191 | controller | - ' ' 2026-03-09 19:10:03.759195 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.759199 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.759204 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.759208 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.759212 | controller | --timeout=300s' 2026-03-09 19:10:03.759216 | controller | - ' ' 2026-03-09 19:10:03.759221 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.759227 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.759231 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.759236 | controller | - ' ' 2026-03-09 19:10:03.759240 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759244 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759248 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759253 | controller | - ' ' 2026-03-09 19:10:03.759258 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.759263 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.759267 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.759271 | controller | available"' 2026-03-09 19:10:03.759276 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.759280 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.759284 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.759288 | controller | - ' exit 1' 2026-03-09 19:10:03.759292 | controller | - ' fi' 2026-03-09 19:10:03.759297 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.759301 | controller | - ' ' 2026-03-09 19:10:03.759305 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.759310 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.759322 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759326 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.759331 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759335 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.759339 | controller | -- \' 2026-03-09 19:10:03.759344 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759348 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.759352 | controller | - ' logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.759357 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.759361 | controller | - ' logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.759365 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.759369 | controller | found' 2026-03-09 19:10:03.759374 | controller | - ' logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.759378 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.759382 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.759386 | controller | - ' ' 2026-03-09 19:10:03.759391 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.759395 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.759399 | controller | - ' ' 2026-03-09 19:10:03.759403 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.759408 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.759414 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.759418 | controller | - ' exit 1' 2026-03-09 19:10:03.759422 | controller | - ' fi' 2026-03-09 19:10:03.759426 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.759431 | controller | - ' ' 2026-03-09 19:10:03.759435 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.759439 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.759444 | controller | | base64 -d)' 2026-03-09 19:10:03.759448 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.759452 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.759456 | controller | - ' exit 1' 2026-03-09 19:10:03.759461 | controller | - ' fi' 2026-03-09 19:10:03.759465 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.759469 | controller | - ' ' 2026-03-09 19:10:03.759473 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.759478 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.759482 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.759486 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.759490 | controller | --timeout=300s' 2026-03-09 19:10:03.759495 | controller | - ' ' 2026-03-09 19:10:03.759499 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.759503 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.759507 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.759512 | controller | - ' ' 2026-03-09 19:10:03.759516 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759520 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759524 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759529 | controller | - ' ' 2026-03-09 19:10:03.759533 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.759537 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.759541 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.759546 | controller | available"' 2026-03-09 19:10:03.759550 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.759554 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.759559 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.759563 | controller | - ' exit 1' 2026-03-09 19:10:03.759567 | controller | - ' fi' 2026-03-09 19:10:03.759571 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.759576 | controller | - ' ' 2026-03-09 19:10:03.759580 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.759584 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.759588 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759600 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.759604 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759610 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.759616 | controller | -- \' 2026-03-09 19:10:03.759621 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759625 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.759629 | controller | - ' logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.759633 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.759638 | controller | - ' logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.759642 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.759646 | controller | found' 2026-03-09 19:10:03.759650 | controller | - ' logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.759655 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.759659 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.759663 | controller | - ' ' 2026-03-09 19:10:03.759686 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.759691 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.759695 | controller | - ' ' 2026-03-09 19:10:03.759699 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.759704 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.759708 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.759712 | controller | - ' exit 1' 2026-03-09 19:10:03.759717 | controller | - ' fi' 2026-03-09 19:10:03.759721 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.759725 | controller | - ' ' 2026-03-09 19:10:03.759729 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.759734 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.759738 | controller | | base64 -d)' 2026-03-09 19:10:03.759742 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.759746 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.759750 | controller | - ' exit 1' 2026-03-09 19:10:03.759755 | controller | - ' fi' 2026-03-09 19:10:03.759759 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.759763 | controller | - ' ' 2026-03-09 19:10:03.759767 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.759772 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.759776 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.759780 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.759784 | controller | --timeout=300s' 2026-03-09 19:10:03.759789 | controller | - ' ' 2026-03-09 19:10:03.759793 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.759797 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.759801 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.759806 | controller | - ' ' 2026-03-09 19:10:03.759810 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759814 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759820 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.759824 | controller | - ' ' 2026-03-09 19:10:03.759829 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.759833 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.759850 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.759855 | controller | available"' 2026-03-09 19:10:03.759859 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.759863 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.759867 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.759872 | controller | - ' exit 1' 2026-03-09 19:10:03.759876 | controller | - ' fi' 2026-03-09 19:10:03.759880 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.759884 | controller | - ' ' 2026-03-09 19:10:03.759889 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.759893 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.759897 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759909 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.759914 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759918 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.759923 | controller | -- \' 2026-03-09 19:10:03.759927 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.759931 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.759935 | controller | - ' logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.759940 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.759944 | controller | - ' logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.759948 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.759952 | controller | found' 2026-03-09 19:10:03.759957 | controller | - ' logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.759961 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.759965 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.759969 | controller | - ' ' 2026-03-09 19:10:03.759974 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.759978 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.759982 | controller | - ' ' 2026-03-09 19:10:03.759986 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.759991 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.759995 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.759999 | controller | - ' exit 1' 2026-03-09 19:10:03.760003 | controller | - ' fi' 2026-03-09 19:10:03.760009 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.760015 | controller | - ' ' 2026-03-09 19:10:03.760022 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.760028 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.760034 | controller | | base64 -d)' 2026-03-09 19:10:03.760043 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.760047 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.760052 | controller | - ' exit 1' 2026-03-09 19:10:03.760057 | controller | - ' fi' 2026-03-09 19:10:03.760061 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.760065 | controller | - ' ' 2026-03-09 19:10:03.760069 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.760074 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.760078 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.760082 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.760086 | controller | --timeout=300s' 2026-03-09 19:10:03.760091 | controller | - ' ' 2026-03-09 19:10:03.760095 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.760099 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.760103 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.760108 | controller | - ' ' 2026-03-09 19:10:03.760112 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760116 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760120 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760127 | controller | - ' ' 2026-03-09 19:10:03.760131 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.760135 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.760140 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.760155 | controller | available"' 2026-03-09 19:10:03.760163 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.760167 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.760172 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.760176 | controller | - ' exit 1' 2026-03-09 19:10:03.760180 | controller | - ' fi' 2026-03-09 19:10:03.760184 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.760189 | controller | - ' ' 2026-03-09 19:10:03.760193 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.760197 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.760201 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760206 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.760218 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760223 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.760227 | controller | -- \' 2026-03-09 19:10:03.760231 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760235 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.760240 | controller | - ' logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.760246 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.760250 | controller | - ' logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.760255 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.760259 | controller | found' 2026-03-09 19:10:03.760263 | controller | - ' logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.760267 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.760272 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.760276 | controller | - ' ' 2026-03-09 19:10:03.760280 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.760285 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.760289 | controller | - ' ' 2026-03-09 19:10:03.760295 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.760299 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.760304 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.760308 | controller | - ' exit 1' 2026-03-09 19:10:03.760312 | controller | - ' fi' 2026-03-09 19:10:03.760316 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.760321 | controller | - ' ' 2026-03-09 19:10:03.760327 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.760332 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.760336 | controller | | base64 -d)' 2026-03-09 19:10:03.760340 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.760346 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.760351 | controller | - ' exit 1' 2026-03-09 19:10:03.760355 | controller | - ' fi' 2026-03-09 19:10:03.760359 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.760363 | controller | - ' ' 2026-03-09 19:10:03.760368 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.760372 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.760376 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.760380 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.760385 | controller | --timeout=300s' 2026-03-09 19:10:03.760389 | controller | - ' ' 2026-03-09 19:10:03.760393 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.760397 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.760402 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.760406 | controller | - ' ' 2026-03-09 19:10:03.760410 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760414 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760419 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760423 | controller | - ' ' 2026-03-09 19:10:03.760427 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.760431 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.760436 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.760440 | controller | available"' 2026-03-09 19:10:03.760446 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.760451 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.760456 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.760460 | controller | - ' exit 1' 2026-03-09 19:10:03.760464 | controller | - ' fi' 2026-03-09 19:10:03.760468 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.760473 | controller | - ' ' 2026-03-09 19:10:03.760477 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.760481 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.760485 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760490 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.760502 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760506 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.760510 | controller | -- \' 2026-03-09 19:10:03.760515 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760519 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.760523 | controller | - ' logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.760528 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.760532 | controller | - ' logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.760536 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.760540 | controller | found' 2026-03-09 19:10:03.760544 | controller | - ' logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.760549 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.760553 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.760557 | controller | - ' ' 2026-03-09 19:10:03.760562 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.760566 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.760570 | controller | - ' ' 2026-03-09 19:10:03.760574 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.760579 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.760583 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.760587 | controller | - ' exit 1' 2026-03-09 19:10:03.760591 | controller | - ' fi' 2026-03-09 19:10:03.760595 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.760600 | controller | - ' ' 2026-03-09 19:10:03.760604 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.760608 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.760612 | controller | | base64 -d)' 2026-03-09 19:10:03.760617 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.760621 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.760625 | controller | - ' exit 1' 2026-03-09 19:10:03.760629 | controller | - ' fi' 2026-03-09 19:10:03.760634 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.760638 | controller | - ' ' 2026-03-09 19:10:03.760642 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.760646 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.760653 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.760657 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.760661 | controller | --timeout=300s' 2026-03-09 19:10:03.760666 | controller | - ' ' 2026-03-09 19:10:03.760670 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.760674 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.760678 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.760682 | controller | - ' ' 2026-03-09 19:10:03.760687 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760691 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760697 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760701 | controller | - ' ' 2026-03-09 19:10:03.760706 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.760710 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.760714 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.760718 | controller | available"' 2026-03-09 19:10:03.760723 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.760727 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.760731 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.760735 | controller | - ' exit 1' 2026-03-09 19:10:03.760740 | controller | - ' fi' 2026-03-09 19:10:03.760744 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.760748 | controller | - ' ' 2026-03-09 19:10:03.760753 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.760757 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.760761 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760765 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.760770 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760781 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.760786 | controller | -- \' 2026-03-09 19:10:03.760790 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.760794 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.760798 | controller | - ' logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.760803 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.760807 | controller | - ' logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.760811 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.760816 | controller | found' 2026-03-09 19:10:03.760820 | controller | - ' logger.go:42: 18:56:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.760824 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.760829 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.760834 | controller | - ' ' 2026-03-09 19:10:03.760839 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.760843 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.760847 | controller | - ' ' 2026-03-09 19:10:03.760851 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.760856 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.760860 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.760864 | controller | - ' exit 1' 2026-03-09 19:10:03.760869 | controller | - ' fi' 2026-03-09 19:10:03.760873 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.760877 | controller | - ' ' 2026-03-09 19:10:03.760881 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.760886 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.760890 | controller | | base64 -d)' 2026-03-09 19:10:03.760894 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.760898 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.760903 | controller | - ' exit 1' 2026-03-09 19:10:03.760907 | controller | - ' fi' 2026-03-09 19:10:03.760911 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.760915 | controller | - ' ' 2026-03-09 19:10:03.760920 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.760924 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.760928 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.760933 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.760937 | controller | --timeout=300s' 2026-03-09 19:10:03.760941 | controller | - ' ' 2026-03-09 19:10:03.760945 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.760950 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.760954 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.760958 | controller | - ' ' 2026-03-09 19:10:03.760962 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760967 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760971 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.760975 | controller | - ' ' 2026-03-09 19:10:03.760979 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.760984 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.760988 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.760992 | controller | available"' 2026-03-09 19:10:03.760996 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.761001 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.761005 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.761011 | controller | - ' exit 1' 2026-03-09 19:10:03.761017 | controller | - ' fi' 2026-03-09 19:10:03.761023 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.761029 | controller | - ' ' 2026-03-09 19:10:03.761038 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.761045 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.761049 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761053 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.761057 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761069 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.761074 | controller | -- \' 2026-03-09 19:10:03.761078 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761082 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.761087 | controller | - ' logger.go:42: 18:56:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.761091 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.761095 | controller | - ' logger.go:42: 18:56:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.761100 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.761104 | controller | found' 2026-03-09 19:10:03.761108 | controller | - ' logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.761112 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.761117 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.761121 | controller | - ' ' 2026-03-09 19:10:03.761125 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.761129 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.761134 | controller | - ' ' 2026-03-09 19:10:03.761138 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.761142 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.761191 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.761196 | controller | - ' exit 1' 2026-03-09 19:10:03.761200 | controller | - ' fi' 2026-03-09 19:10:03.761204 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.761209 | controller | - ' ' 2026-03-09 19:10:03.761213 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.761217 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.761221 | controller | | base64 -d)' 2026-03-09 19:10:03.761226 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.761230 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.761234 | controller | - ' exit 1' 2026-03-09 19:10:03.761238 | controller | - ' fi' 2026-03-09 19:10:03.761243 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.761247 | controller | - ' ' 2026-03-09 19:10:03.761251 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.761255 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.761260 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.761264 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.761268 | controller | --timeout=300s' 2026-03-09 19:10:03.761272 | controller | - ' ' 2026-03-09 19:10:03.761277 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.761281 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.761287 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.761292 | controller | - ' ' 2026-03-09 19:10:03.761296 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761300 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761304 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761309 | controller | - ' ' 2026-03-09 19:10:03.761313 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.761317 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.761322 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.761326 | controller | available"' 2026-03-09 19:10:03.761332 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.761337 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.761341 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.761345 | controller | - ' exit 1' 2026-03-09 19:10:03.761349 | controller | - ' fi' 2026-03-09 19:10:03.761354 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.761358 | controller | - ' ' 2026-03-09 19:10:03.761362 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.761366 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.761371 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761375 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.761379 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761383 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.761388 | controller | -- \' 2026-03-09 19:10:03.761400 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761405 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.761410 | controller | - ' logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.761414 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.761418 | controller | - ' logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.761422 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.761427 | controller | found' 2026-03-09 19:10:03.761431 | controller | - ' logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.761435 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.761439 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.761444 | controller | - ' ' 2026-03-09 19:10:03.761448 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.761452 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.761456 | controller | - ' ' 2026-03-09 19:10:03.761461 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.761465 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.761469 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.761477 | controller | - ' exit 1' 2026-03-09 19:10:03.761482 | controller | - ' fi' 2026-03-09 19:10:03.761486 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.761490 | controller | - ' ' 2026-03-09 19:10:03.761494 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.761499 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.761503 | controller | | base64 -d)' 2026-03-09 19:10:03.761507 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.761511 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.761516 | controller | - ' exit 1' 2026-03-09 19:10:03.761520 | controller | - ' fi' 2026-03-09 19:10:03.761524 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.761528 | controller | - ' ' 2026-03-09 19:10:03.761533 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.761537 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.761541 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.761545 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.761549 | controller | --timeout=300s' 2026-03-09 19:10:03.761554 | controller | - ' ' 2026-03-09 19:10:03.761558 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.761562 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.761567 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.761571 | controller | - ' ' 2026-03-09 19:10:03.761575 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761579 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761584 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761588 | controller | - ' ' 2026-03-09 19:10:03.761592 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.761596 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.761600 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.761605 | controller | available"' 2026-03-09 19:10:03.761609 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.761615 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.761620 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.761624 | controller | - ' exit 1' 2026-03-09 19:10:03.761628 | controller | - ' fi' 2026-03-09 19:10:03.761632 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.761637 | controller | - ' ' 2026-03-09 19:10:03.761641 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.761645 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.761649 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761655 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.761660 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761666 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.761670 | controller | -- \' 2026-03-09 19:10:03.761681 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761686 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.761691 | controller | - ' logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.761696 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.761700 | controller | - ' logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.761706 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.761710 | controller | found' 2026-03-09 19:10:03.761715 | controller | - ' logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.761719 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.761723 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.761727 | controller | - ' ' 2026-03-09 19:10:03.761732 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.761736 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.761740 | controller | - ' ' 2026-03-09 19:10:03.761745 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.761749 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.761753 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.761757 | controller | - ' exit 1' 2026-03-09 19:10:03.761761 | controller | - ' fi' 2026-03-09 19:10:03.761766 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.761770 | controller | - ' ' 2026-03-09 19:10:03.761774 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.761778 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.761783 | controller | | base64 -d)' 2026-03-09 19:10:03.761787 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.761791 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.761795 | controller | - ' exit 1' 2026-03-09 19:10:03.761800 | controller | - ' fi' 2026-03-09 19:10:03.761804 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.761808 | controller | - ' ' 2026-03-09 19:10:03.761812 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.761817 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.761821 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.761825 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.761829 | controller | --timeout=300s' 2026-03-09 19:10:03.761833 | controller | - ' ' 2026-03-09 19:10:03.761838 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.761842 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.761846 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.761850 | controller | - ' ' 2026-03-09 19:10:03.761855 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761859 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761863 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.761869 | controller | - ' ' 2026-03-09 19:10:03.761873 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.761878 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.761882 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.761886 | controller | available"' 2026-03-09 19:10:03.761890 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.761895 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.761899 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.761903 | controller | - ' exit 1' 2026-03-09 19:10:03.761909 | controller | - ' fi' 2026-03-09 19:10:03.761913 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.761918 | controller | - ' ' 2026-03-09 19:10:03.761922 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.761926 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.761930 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761935 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.761939 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761943 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.761947 | controller | -- \' 2026-03-09 19:10:03.761959 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.761963 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.761968 | controller | - ' logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.761972 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.761976 | controller | - ' logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.761980 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.761984 | controller | found' 2026-03-09 19:10:03.761989 | controller | - ' logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.761993 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.761997 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.762002 | controller | - ' ' 2026-03-09 19:10:03.762007 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.762013 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.762019 | controller | - ' ' 2026-03-09 19:10:03.762025 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.762031 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.762037 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.762043 | controller | - ' exit 1' 2026-03-09 19:10:03.762047 | controller | - ' fi' 2026-03-09 19:10:03.762054 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.762059 | controller | - ' ' 2026-03-09 19:10:03.762063 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.762067 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.762071 | controller | | base64 -d)' 2026-03-09 19:10:03.762076 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.762082 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.762086 | controller | - ' exit 1' 2026-03-09 19:10:03.762091 | controller | - ' fi' 2026-03-09 19:10:03.762095 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.762100 | controller | - ' ' 2026-03-09 19:10:03.762106 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.762112 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.762117 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.762123 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.762128 | controller | --timeout=300s' 2026-03-09 19:10:03.762134 | controller | - ' ' 2026-03-09 19:10:03.762139 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.762166 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.762177 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.762183 | controller | - ' ' 2026-03-09 19:10:03.762189 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762195 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762201 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762207 | controller | - ' ' 2026-03-09 19:10:03.762213 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.762219 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.762224 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.762230 | controller | available"' 2026-03-09 19:10:03.762236 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.762241 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.762247 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.762253 | controller | - ' exit 1' 2026-03-09 19:10:03.762259 | controller | - ' fi' 2026-03-09 19:10:03.762265 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.762270 | controller | - ' ' 2026-03-09 19:10:03.762275 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.762279 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.762283 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762288 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.762292 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762296 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.762301 | controller | -- \' 2026-03-09 19:10:03.762305 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762323 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.762329 | controller | - ' logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.762333 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.762340 | controller | - ' logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.762344 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.762348 | controller | found' 2026-03-09 19:10:03.762353 | controller | - ' logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.762357 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.762361 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.762366 | controller | - ' ' 2026-03-09 19:10:03.762370 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.762374 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.762378 | controller | - ' ' 2026-03-09 19:10:03.762383 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.762387 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.762391 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.762396 | controller | - ' exit 1' 2026-03-09 19:10:03.762400 | controller | - ' fi' 2026-03-09 19:10:03.762404 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.762408 | controller | - ' ' 2026-03-09 19:10:03.762413 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.762417 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.762421 | controller | | base64 -d)' 2026-03-09 19:10:03.762425 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.762430 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.762434 | controller | - ' exit 1' 2026-03-09 19:10:03.762438 | controller | - ' fi' 2026-03-09 19:10:03.762442 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.762447 | controller | - ' ' 2026-03-09 19:10:03.762453 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.762457 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.762461 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.762466 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.762470 | controller | --timeout=300s' 2026-03-09 19:10:03.762474 | controller | - ' ' 2026-03-09 19:10:03.762479 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.762483 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.762487 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.762491 | controller | - ' ' 2026-03-09 19:10:03.762496 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762500 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762504 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762508 | controller | - ' ' 2026-03-09 19:10:03.762513 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.762517 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.762521 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.762525 | controller | available"' 2026-03-09 19:10:03.762530 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.762535 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.762540 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.762544 | controller | - ' exit 1' 2026-03-09 19:10:03.762548 | controller | - ' fi' 2026-03-09 19:10:03.762552 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.762557 | controller | - ' ' 2026-03-09 19:10:03.762561 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.762565 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.762569 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762574 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.762580 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762584 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.762588 | controller | -- \' 2026-03-09 19:10:03.762593 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762597 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.762609 | controller | - ' logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.762613 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.762617 | controller | - ' logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.762622 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.762626 | controller | found' 2026-03-09 19:10:03.762630 | controller | - ' logger.go:42: 18:56:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.762634 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.762639 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.762643 | controller | - ' ' 2026-03-09 19:10:03.762647 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.762652 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.762656 | controller | - ' ' 2026-03-09 19:10:03.762660 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.762664 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.762669 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.762673 | controller | - ' exit 1' 2026-03-09 19:10:03.762677 | controller | - ' fi' 2026-03-09 19:10:03.762681 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.762686 | controller | - ' ' 2026-03-09 19:10:03.762690 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.762694 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.762699 | controller | | base64 -d)' 2026-03-09 19:10:03.762703 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.762707 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.762711 | controller | - ' exit 1' 2026-03-09 19:10:03.762716 | controller | - ' fi' 2026-03-09 19:10:03.762720 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.762724 | controller | - ' ' 2026-03-09 19:10:03.762728 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.762733 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.762737 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.762745 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.762749 | controller | --timeout=300s' 2026-03-09 19:10:03.762753 | controller | - ' ' 2026-03-09 19:10:03.762757 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.762762 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.762766 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.762770 | controller | - ' ' 2026-03-09 19:10:03.762775 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762779 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762784 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.762790 | controller | - ' ' 2026-03-09 19:10:03.762795 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.762801 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.762806 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.762810 | controller | available"' 2026-03-09 19:10:03.762814 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.762818 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.762823 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.762827 | controller | - ' exit 1' 2026-03-09 19:10:03.762831 | controller | - ' fi' 2026-03-09 19:10:03.762836 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.762840 | controller | - ' ' 2026-03-09 19:10:03.762844 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.762848 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.762853 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762857 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.762861 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762865 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.762870 | controller | -- \' 2026-03-09 19:10:03.762874 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.762878 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.762891 | controller | - ' logger.go:42: 18:56:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.762895 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.762900 | controller | - ' logger.go:42: 18:56:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.762904 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.762908 | controller | found' 2026-03-09 19:10:03.762913 | controller | - ' logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.762917 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.762921 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.762926 | controller | - ' ' 2026-03-09 19:10:03.762930 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.762936 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.762940 | controller | - ' ' 2026-03-09 19:10:03.762945 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.762949 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.762953 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.762958 | controller | - ' exit 1' 2026-03-09 19:10:03.762962 | controller | - ' fi' 2026-03-09 19:10:03.762966 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.762970 | controller | - ' ' 2026-03-09 19:10:03.762977 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.762981 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.762985 | controller | | base64 -d)' 2026-03-09 19:10:03.762989 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.762994 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.762998 | controller | - ' exit 1' 2026-03-09 19:10:03.763002 | controller | - ' fi' 2026-03-09 19:10:03.763007 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.763013 | controller | - ' ' 2026-03-09 19:10:03.763019 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.763026 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.763032 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.763038 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.763042 | controller | --timeout=300s' 2026-03-09 19:10:03.763046 | controller | - ' ' 2026-03-09 19:10:03.763051 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.763055 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.763059 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.763063 | controller | - ' ' 2026-03-09 19:10:03.763070 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763074 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763078 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763083 | controller | - ' ' 2026-03-09 19:10:03.763087 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.763091 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.763095 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.763100 | controller | available"' 2026-03-09 19:10:03.763104 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.763108 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.763112 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.763117 | controller | - ' exit 1' 2026-03-09 19:10:03.763121 | controller | - ' fi' 2026-03-09 19:10:03.763125 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.763129 | controller | - ' ' 2026-03-09 19:10:03.763134 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.763140 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.763157 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763164 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.763169 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763173 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.763177 | controller | -- \' 2026-03-09 19:10:03.763182 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763186 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.763190 | controller | - ' logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.763203 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.763207 | controller | - ' logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.763212 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.763216 | controller | found' 2026-03-09 19:10:03.763220 | controller | - ' logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.763225 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.763229 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.763233 | controller | - ' ' 2026-03-09 19:10:03.763238 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.763242 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.763246 | controller | - ' ' 2026-03-09 19:10:03.763250 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.763255 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.763259 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.763263 | controller | - ' exit 1' 2026-03-09 19:10:03.763267 | controller | - ' fi' 2026-03-09 19:10:03.763272 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.763276 | controller | - ' ' 2026-03-09 19:10:03.763280 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.763285 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.763289 | controller | | base64 -d)' 2026-03-09 19:10:03.763293 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.763297 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.763302 | controller | - ' exit 1' 2026-03-09 19:10:03.763306 | controller | - ' fi' 2026-03-09 19:10:03.763310 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.763314 | controller | - ' ' 2026-03-09 19:10:03.763319 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.763323 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.763327 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.763331 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.763336 | controller | --timeout=300s' 2026-03-09 19:10:03.763340 | controller | - ' ' 2026-03-09 19:10:03.763344 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.763349 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.763355 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.763359 | controller | - ' ' 2026-03-09 19:10:03.763363 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763368 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763372 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763376 | controller | - ' ' 2026-03-09 19:10:03.763380 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.763385 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.763389 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.763393 | controller | available"' 2026-03-09 19:10:03.763398 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.763402 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.763406 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.763410 | controller | - ' exit 1' 2026-03-09 19:10:03.763415 | controller | - ' fi' 2026-03-09 19:10:03.763419 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.763423 | controller | - ' ' 2026-03-09 19:10:03.763428 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.763432 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.763436 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763440 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.763445 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763449 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.763453 | controller | -- \' 2026-03-09 19:10:03.763458 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763463 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.763468 | controller | - ' logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.763473 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.763484 | controller | - ' logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.763491 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.763495 | controller | found' 2026-03-09 19:10:03.763500 | controller | - ' logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.763504 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.763508 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.763513 | controller | - ' ' 2026-03-09 19:10:03.763517 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.763521 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.763526 | controller | - ' ' 2026-03-09 19:10:03.763530 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.763534 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.763539 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.763543 | controller | - ' exit 1' 2026-03-09 19:10:03.763549 | controller | - ' fi' 2026-03-09 19:10:03.763555 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.763559 | controller | - ' ' 2026-03-09 19:10:03.763563 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.763568 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.763572 | controller | | base64 -d)' 2026-03-09 19:10:03.763576 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.763581 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.763585 | controller | - ' exit 1' 2026-03-09 19:10:03.763589 | controller | - ' fi' 2026-03-09 19:10:03.763593 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.763598 | controller | - ' ' 2026-03-09 19:10:03.763602 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.763606 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.763610 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.763614 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.763619 | controller | --timeout=300s' 2026-03-09 19:10:03.763623 | controller | - ' ' 2026-03-09 19:10:03.763627 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.763631 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.763636 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.763640 | controller | - ' ' 2026-03-09 19:10:03.763644 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763648 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763653 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763657 | controller | - ' ' 2026-03-09 19:10:03.763661 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.763680 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.763690 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.763695 | controller | available"' 2026-03-09 19:10:03.763699 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.763703 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.763708 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.763712 | controller | - ' exit 1' 2026-03-09 19:10:03.763716 | controller | - ' fi' 2026-03-09 19:10:03.763720 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.763724 | controller | - ' ' 2026-03-09 19:10:03.763729 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.763733 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.763737 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763741 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.763746 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763750 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.763756 | controller | -- \' 2026-03-09 19:10:03.763760 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.763765 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.763769 | controller | - ' logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.763773 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.763785 | controller | - ' logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.763790 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.763794 | controller | found' 2026-03-09 19:10:03.763799 | controller | - ' logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.763803 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.763807 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.763811 | controller | - ' ' 2026-03-09 19:10:03.763816 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.763820 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.763824 | controller | - ' ' 2026-03-09 19:10:03.763829 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.763833 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.763837 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.763841 | controller | - ' exit 1' 2026-03-09 19:10:03.763846 | controller | - ' fi' 2026-03-09 19:10:03.763850 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.763854 | controller | - ' ' 2026-03-09 19:10:03.763858 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.763863 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.763867 | controller | | base64 -d)' 2026-03-09 19:10:03.763871 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.763876 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.763880 | controller | - ' exit 1' 2026-03-09 19:10:03.763884 | controller | - ' fi' 2026-03-09 19:10:03.763888 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.763893 | controller | - ' ' 2026-03-09 19:10:03.763897 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.763901 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.763905 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.763910 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.763914 | controller | --timeout=300s' 2026-03-09 19:10:03.763918 | controller | - ' ' 2026-03-09 19:10:03.763922 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.763927 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.763933 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.763937 | controller | - ' ' 2026-03-09 19:10:03.763942 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763946 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763950 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.763956 | controller | - ' ' 2026-03-09 19:10:03.763960 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.763964 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.763969 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.763973 | controller | available"' 2026-03-09 19:10:03.763977 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.763982 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.763986 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.763990 | controller | - ' exit 1' 2026-03-09 19:10:03.763994 | controller | - ' fi' 2026-03-09 19:10:03.763999 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.764003 | controller | - ' ' 2026-03-09 19:10:03.764008 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.764014 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.764020 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764026 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.764032 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764038 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.764043 | controller | -- \' 2026-03-09 19:10:03.764047 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764051 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.764055 | controller | - ' logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.764060 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.764064 | controller | - ' logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.764076 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.764081 | controller | found' 2026-03-09 19:10:03.764087 | controller | - ' logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.764092 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.764096 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.764100 | controller | - ' ' 2026-03-09 19:10:03.764104 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.764109 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.764113 | controller | - ' ' 2026-03-09 19:10:03.764117 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.764122 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.764126 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.764130 | controller | - ' exit 1' 2026-03-09 19:10:03.764134 | controller | - ' fi' 2026-03-09 19:10:03.764139 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.764143 | controller | - ' ' 2026-03-09 19:10:03.764162 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.764166 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.764170 | controller | | base64 -d)' 2026-03-09 19:10:03.764175 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.764179 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.764187 | controller | - ' exit 1' 2026-03-09 19:10:03.764192 | controller | - ' fi' 2026-03-09 19:10:03.764196 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.764200 | controller | - ' ' 2026-03-09 19:10:03.764205 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.764209 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.764213 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.764217 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.764222 | controller | --timeout=300s' 2026-03-09 19:10:03.764226 | controller | - ' ' 2026-03-09 19:10:03.764230 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.764234 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.764239 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.764243 | controller | - ' ' 2026-03-09 19:10:03.764247 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764251 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764256 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764260 | controller | - ' ' 2026-03-09 19:10:03.764264 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.764269 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.764273 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.764277 | controller | available"' 2026-03-09 19:10:03.764281 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.764287 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.764292 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.764296 | controller | - ' exit 1' 2026-03-09 19:10:03.764300 | controller | - ' fi' 2026-03-09 19:10:03.764305 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.764309 | controller | - ' ' 2026-03-09 19:10:03.764313 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.764317 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.764322 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764326 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.764330 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764334 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.764339 | controller | -- \' 2026-03-09 19:10:03.764343 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764347 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.764352 | controller | - ' logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.764356 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.764360 | controller | - ' logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.764366 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.764378 | controller | found' 2026-03-09 19:10:03.764382 | controller | - ' logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.764387 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.764391 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.764395 | controller | - ' ' 2026-03-09 19:10:03.764400 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.764404 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.764408 | controller | - ' ' 2026-03-09 19:10:03.764413 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.764417 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.764421 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.764426 | controller | - ' exit 1' 2026-03-09 19:10:03.764430 | controller | - ' fi' 2026-03-09 19:10:03.764434 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.764439 | controller | - ' ' 2026-03-09 19:10:03.764443 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.764447 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.764451 | controller | | base64 -d)' 2026-03-09 19:10:03.764456 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.764462 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.764466 | controller | - ' exit 1' 2026-03-09 19:10:03.764470 | controller | - ' fi' 2026-03-09 19:10:03.764475 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.764479 | controller | - ' ' 2026-03-09 19:10:03.764483 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.764488 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.764492 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.764496 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.764500 | controller | --timeout=300s' 2026-03-09 19:10:03.764505 | controller | - ' ' 2026-03-09 19:10:03.764509 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.764513 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.764517 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.764522 | controller | - ' ' 2026-03-09 19:10:03.764526 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764530 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764534 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764539 | controller | - ' ' 2026-03-09 19:10:03.764543 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.764547 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.764551 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.764556 | controller | available"' 2026-03-09 19:10:03.764560 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.764564 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.764570 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.764575 | controller | - ' exit 1' 2026-03-09 19:10:03.764579 | controller | - ' fi' 2026-03-09 19:10:03.764585 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.764589 | controller | - ' ' 2026-03-09 19:10:03.764593 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.764598 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.764602 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764606 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.764610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764615 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.764619 | controller | -- \' 2026-03-09 19:10:03.764623 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764627 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.764632 | controller | - ' logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.764636 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.764640 | controller | - ' logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.764644 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.764649 | controller | found' 2026-03-09 19:10:03.764659 | controller | - ' logger.go:42: 18:56:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.764664 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.764668 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.764672 | controller | - ' ' 2026-03-09 19:10:03.764678 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.764682 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.764687 | controller | - ' ' 2026-03-09 19:10:03.764691 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.764695 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.764700 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.764704 | controller | - ' exit 1' 2026-03-09 19:10:03.764708 | controller | - ' fi' 2026-03-09 19:10:03.764713 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.764717 | controller | - ' ' 2026-03-09 19:10:03.764721 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.764725 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.764730 | controller | | base64 -d)' 2026-03-09 19:10:03.764734 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.764738 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.764742 | controller | - ' exit 1' 2026-03-09 19:10:03.764747 | controller | - ' fi' 2026-03-09 19:10:03.764751 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.764755 | controller | - ' ' 2026-03-09 19:10:03.764759 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.764764 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.764768 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.764774 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.764778 | controller | --timeout=300s' 2026-03-09 19:10:03.764782 | controller | - ' ' 2026-03-09 19:10:03.764787 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.764791 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.764795 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.764799 | controller | - ' ' 2026-03-09 19:10:03.764804 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764808 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764812 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.764816 | controller | - ' ' 2026-03-09 19:10:03.764821 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.764825 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.764829 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.764833 | controller | available"' 2026-03-09 19:10:03.764838 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.764842 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.764846 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.764850 | controller | - ' exit 1' 2026-03-09 19:10:03.764855 | controller | - ' fi' 2026-03-09 19:10:03.764859 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.764863 | controller | - ' ' 2026-03-09 19:10:03.764868 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.764872 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.764876 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764880 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.764886 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764890 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.764895 | controller | -- \' 2026-03-09 19:10:03.764899 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.764903 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.764908 | controller | - ' logger.go:42: 18:56:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.764912 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.764916 | controller | - ' logger.go:42: 18:56:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.764920 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.764925 | controller | found' 2026-03-09 19:10:03.764929 | controller | - ' logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.764940 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.764944 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.764948 | controller | - ' ' 2026-03-09 19:10:03.764953 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.764959 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.764963 | controller | - ' ' 2026-03-09 19:10:03.764967 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.764971 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.764976 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.764980 | controller | - ' exit 1' 2026-03-09 19:10:03.764984 | controller | - ' fi' 2026-03-09 19:10:03.764989 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.764993 | controller | - ' ' 2026-03-09 19:10:03.764997 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.765001 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.765007 | controller | | base64 -d)' 2026-03-09 19:10:03.765012 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.765018 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.765024 | controller | - ' exit 1' 2026-03-09 19:10:03.765030 | controller | - ' fi' 2026-03-09 19:10:03.765036 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.765042 | controller | - ' ' 2026-03-09 19:10:03.765048 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.765053 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.765059 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.765064 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.765070 | controller | --timeout=300s' 2026-03-09 19:10:03.765076 | controller | - ' ' 2026-03-09 19:10:03.765081 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.765087 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.765093 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.765099 | controller | - ' ' 2026-03-09 19:10:03.765104 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765110 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765116 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765122 | controller | - ' ' 2026-03-09 19:10:03.765127 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.765133 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.765139 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.765159 | controller | available"' 2026-03-09 19:10:03.765169 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.765176 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.765181 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.765187 | controller | - ' exit 1' 2026-03-09 19:10:03.765193 | controller | - ' fi' 2026-03-09 19:10:03.765199 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.765204 | controller | - ' ' 2026-03-09 19:10:03.765209 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.765215 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.765224 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765230 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.765236 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765242 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.765248 | controller | -- \' 2026-03-09 19:10:03.765254 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765259 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.765265 | controller | - ' logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.765270 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.765276 | controller | - ' logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.765282 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.765287 | controller | found' 2026-03-09 19:10:03.765292 | controller | - ' logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.765298 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.765318 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.765325 | controller | - ' ' 2026-03-09 19:10:03.765330 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.765339 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.765345 | controller | - ' ' 2026-03-09 19:10:03.765350 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.765355 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.765361 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.765367 | controller | - ' exit 1' 2026-03-09 19:10:03.765372 | controller | - ' fi' 2026-03-09 19:10:03.765378 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.765383 | controller | - ' ' 2026-03-09 19:10:03.765389 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.765394 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.765400 | controller | | base64 -d)' 2026-03-09 19:10:03.765405 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.765411 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.765416 | controller | - ' exit 1' 2026-03-09 19:10:03.765421 | controller | - ' fi' 2026-03-09 19:10:03.765427 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.765433 | controller | - ' ' 2026-03-09 19:10:03.765438 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.765444 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.765449 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.765454 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.765460 | controller | --timeout=300s' 2026-03-09 19:10:03.765465 | controller | - ' ' 2026-03-09 19:10:03.765471 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.765476 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.765481 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.765489 | controller | - ' ' 2026-03-09 19:10:03.765495 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765500 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765505 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765510 | controller | - ' ' 2026-03-09 19:10:03.765516 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.765521 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.765526 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.765532 | controller | available"' 2026-03-09 19:10:03.765537 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.765542 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.765547 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.765553 | controller | - ' exit 1' 2026-03-09 19:10:03.765559 | controller | - ' fi' 2026-03-09 19:10:03.765564 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.765569 | controller | - ' ' 2026-03-09 19:10:03.765574 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.765579 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.765584 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765589 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.765594 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765600 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.765605 | controller | -- \' 2026-03-09 19:10:03.765610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765615 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.765620 | controller | - ' logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.765625 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.765630 | controller | - ' logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.765635 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.765640 | controller | found' 2026-03-09 19:10:03.765645 | controller | - ' logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.765650 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.765655 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.765660 | controller | - ' ' 2026-03-09 19:10:03.765675 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.765681 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.765686 | controller | - ' ' 2026-03-09 19:10:03.765691 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.765696 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.765701 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.765706 | controller | - ' exit 1' 2026-03-09 19:10:03.765711 | controller | - ' fi' 2026-03-09 19:10:03.765716 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.765725 | controller | - ' ' 2026-03-09 19:10:03.765733 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.765738 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.765744 | controller | | base64 -d)' 2026-03-09 19:10:03.765749 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.765754 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.765760 | controller | - ' exit 1' 2026-03-09 19:10:03.765769 | controller | - ' fi' 2026-03-09 19:10:03.765774 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.765779 | controller | - ' ' 2026-03-09 19:10:03.765785 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.765790 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.765795 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.765800 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.765805 | controller | --timeout=300s' 2026-03-09 19:10:03.765810 | controller | - ' ' 2026-03-09 19:10:03.765817 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.765822 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.765827 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.765832 | controller | - ' ' 2026-03-09 19:10:03.765838 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765843 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765847 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.765853 | controller | - ' ' 2026-03-09 19:10:03.765858 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.765863 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.765868 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.765874 | controller | available"' 2026-03-09 19:10:03.765879 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.765884 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.765889 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.765894 | controller | - ' exit 1' 2026-03-09 19:10:03.765899 | controller | - ' fi' 2026-03-09 19:10:03.765905 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.765910 | controller | - ' ' 2026-03-09 19:10:03.765915 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.765920 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.765926 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765932 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.765937 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765942 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.765947 | controller | -- \' 2026-03-09 19:10:03.765962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.765967 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.765975 | controller | - ' logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.765981 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.765988 | controller | - ' logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.765993 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.765999 | controller | found' 2026-03-09 19:10:03.766004 | controller | - ' logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.766010 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.766015 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.766021 | controller | - ' ' 2026-03-09 19:10:03.766036 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.766043 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.766048 | controller | - ' ' 2026-03-09 19:10:03.766056 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.766061 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.766066 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.766071 | controller | - ' exit 1' 2026-03-09 19:10:03.766076 | controller | - ' fi' 2026-03-09 19:10:03.766081 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.766086 | controller | - ' ' 2026-03-09 19:10:03.766091 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.766096 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.766101 | controller | | base64 -d)' 2026-03-09 19:10:03.766106 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.766112 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.766117 | controller | - ' exit 1' 2026-03-09 19:10:03.766122 | controller | - ' fi' 2026-03-09 19:10:03.766128 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.766133 | controller | - ' ' 2026-03-09 19:10:03.766138 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.766143 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.766165 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.766171 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.766176 | controller | --timeout=300s' 2026-03-09 19:10:03.766185 | controller | - ' ' 2026-03-09 19:10:03.766191 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.766196 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.766202 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.766208 | controller | - ' ' 2026-03-09 19:10:03.766213 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766218 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766223 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766229 | controller | - ' ' 2026-03-09 19:10:03.766236 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.766242 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.766247 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.766252 | controller | available"' 2026-03-09 19:10:03.766258 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.766264 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.766269 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.766274 | controller | - ' exit 1' 2026-03-09 19:10:03.766283 | controller | - ' fi' 2026-03-09 19:10:03.766289 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.766295 | controller | - ' ' 2026-03-09 19:10:03.766300 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.766306 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.766312 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.766318 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.766323 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.766329 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.766333 | controller | -- \' 2026-03-09 19:10:03.766338 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.766342 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.766346 | controller | - ' logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.766351 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.766355 | controller | - ' logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.766359 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.766363 | controller | found' 2026-03-09 19:10:03.766368 | controller | - ' logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.766372 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.766376 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.766381 | controller | - ' ' 2026-03-09 19:10:03.766385 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.766399 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.766404 | controller | - ' ' 2026-03-09 19:10:03.766408 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.766413 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.766417 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.766421 | controller | - ' exit 1' 2026-03-09 19:10:03.766426 | controller | - ' fi' 2026-03-09 19:10:03.766430 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.766434 | controller | - ' ' 2026-03-09 19:10:03.766439 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.766443 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.766447 | controller | | base64 -d)' 2026-03-09 19:10:03.766451 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.766456 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.766463 | controller | - ' exit 1' 2026-03-09 19:10:03.766467 | controller | - ' fi' 2026-03-09 19:10:03.766471 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.766475 | controller | - ' ' 2026-03-09 19:10:03.766480 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.766484 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.766488 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.766493 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.766497 | controller | --timeout=300s' 2026-03-09 19:10:03.766501 | controller | - ' ' 2026-03-09 19:10:03.766505 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.766510 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.766514 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.766518 | controller | - ' ' 2026-03-09 19:10:03.766523 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766529 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766534 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766540 | controller | - ' ' 2026-03-09 19:10:03.766545 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.766552 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.766557 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.766562 | controller | available"' 2026-03-09 19:10:03.766567 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.766572 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.766578 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.766583 | controller | - ' exit 1' 2026-03-09 19:10:03.766589 | controller | - ' fi' 2026-03-09 19:10:03.766593 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.766599 | controller | - ' ' 2026-03-09 19:10:03.766604 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.766610 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.766615 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.766620 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.766625 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.766631 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.766636 | controller | -- \' 2026-03-09 19:10:03.766643 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.766648 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.766654 | controller | - ' logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.766659 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.766665 | controller | - ' logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.766674 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.766680 | controller | found' 2026-03-09 19:10:03.766686 | controller | - ' logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.766692 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.766698 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.766703 | controller | - ' ' 2026-03-09 19:10:03.766709 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.766715 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.766730 | controller | - ' ' 2026-03-09 19:10:03.766736 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.766741 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.766746 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.766750 | controller | - ' exit 1' 2026-03-09 19:10:03.766756 | controller | - ' fi' 2026-03-09 19:10:03.766762 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.766771 | controller | - ' ' 2026-03-09 19:10:03.766777 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.766783 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.766788 | controller | | base64 -d)' 2026-03-09 19:10:03.766794 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.766800 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.766806 | controller | - ' exit 1' 2026-03-09 19:10:03.766812 | controller | - ' fi' 2026-03-09 19:10:03.766818 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.766823 | controller | - ' ' 2026-03-09 19:10:03.766829 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.766835 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.766841 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.766847 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.766853 | controller | --timeout=300s' 2026-03-09 19:10:03.766859 | controller | - ' ' 2026-03-09 19:10:03.766865 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.766871 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.766877 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.766883 | controller | - ' ' 2026-03-09 19:10:03.766888 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766894 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766899 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.766905 | controller | - ' ' 2026-03-09 19:10:03.766911 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.766917 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.766923 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.766928 | controller | available"' 2026-03-09 19:10:03.766934 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.766940 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.766945 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.766954 | controller | - ' exit 1' 2026-03-09 19:10:03.766960 | controller | - ' fi' 2026-03-09 19:10:03.766966 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.766971 | controller | - ' ' 2026-03-09 19:10:03.766977 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.766982 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.766987 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.766992 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.766998 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.767003 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.767009 | controller | -- \' 2026-03-09 19:10:03.767015 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.767020 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.767026 | controller | - ' logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.767032 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.767038 | controller | - ' logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.767044 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.767049 | controller | found' 2026-03-09 19:10:03.767055 | controller | - ' logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.767060 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.767066 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.767072 | controller | - ' ' 2026-03-09 19:10:03.767077 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.767081 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.767085 | controller | - ' ' 2026-03-09 19:10:03.767099 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.767104 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.767108 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.767112 | controller | - ' exit 1' 2026-03-09 19:10:03.767117 | controller | - ' fi' 2026-03-09 19:10:03.767121 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.767125 | controller | - ' ' 2026-03-09 19:10:03.767130 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.767134 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.767138 | controller | | base64 -d)' 2026-03-09 19:10:03.767193 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.767201 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.767205 | controller | - ' exit 1' 2026-03-09 19:10:03.767210 | controller | - ' fi' 2026-03-09 19:10:03.767214 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.767218 | controller | - ' ' 2026-03-09 19:10:03.767222 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.767227 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.767231 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.767235 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.767242 | controller | --timeout=300s' 2026-03-09 19:10:03.767246 | controller | - ' ' 2026-03-09 19:10:03.767251 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.767255 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.767259 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.767263 | controller | - ' ' 2026-03-09 19:10:03.767268 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767272 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767276 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767280 | controller | - ' ' 2026-03-09 19:10:03.767285 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.767289 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.767300 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.767305 | controller | available"' 2026-03-09 19:10:03.767309 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.767318 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.767324 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.767330 | controller | - ' exit 1' 2026-03-09 19:10:03.767335 | controller | - ' fi' 2026-03-09 19:10:03.767341 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.767346 | controller | - ' ' 2026-03-09 19:10:03.767352 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.767357 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.767363 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.767368 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.767374 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.767379 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.767388 | controller | -- \' 2026-03-09 19:10:03.767394 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.767400 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.767405 | controller | - ' logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.767411 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.767416 | controller | - ' logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.767422 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.767428 | controller | found' 2026-03-09 19:10:03.767434 | controller | - ' logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.767439 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.767445 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.767451 | controller | - ' ' 2026-03-09 19:10:03.767456 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.767462 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.767470 | controller | - ' ' 2026-03-09 19:10:03.767476 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.767493 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.767498 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.767503 | controller | - ' exit 1' 2026-03-09 19:10:03.767507 | controller | - ' fi' 2026-03-09 19:10:03.767511 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.767516 | controller | - ' ' 2026-03-09 19:10:03.767520 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.767524 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.767528 | controller | | base64 -d)' 2026-03-09 19:10:03.767534 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.767540 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.767546 | controller | - ' exit 1' 2026-03-09 19:10:03.767552 | controller | - ' fi' 2026-03-09 19:10:03.767558 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.767564 | controller | - ' ' 2026-03-09 19:10:03.767569 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.767574 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.767581 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.767586 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.767590 | controller | --timeout=300s' 2026-03-09 19:10:03.767594 | controller | - ' ' 2026-03-09 19:10:03.767599 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.767603 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.767607 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.767611 | controller | - ' ' 2026-03-09 19:10:03.767617 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767621 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767626 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767630 | controller | - ' ' 2026-03-09 19:10:03.767634 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.767639 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.767643 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.767647 | controller | available"' 2026-03-09 19:10:03.767651 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.767656 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.767683 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.767688 | controller | - ' exit 1' 2026-03-09 19:10:03.767693 | controller | - ' fi' 2026-03-09 19:10:03.767697 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.767701 | controller | - ' ' 2026-03-09 19:10:03.767705 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.767710 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.767723 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.767730 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.767734 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.767739 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.767743 | controller | -- \' 2026-03-09 19:10:03.767747 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.767751 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.767756 | controller | - ' logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.767760 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.767764 | controller | - ' logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.767769 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.767773 | controller | found' 2026-03-09 19:10:03.767777 | controller | - ' logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.767781 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.767786 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.767790 | controller | - ' ' 2026-03-09 19:10:03.767794 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.767798 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.767803 | controller | - ' ' 2026-03-09 19:10:03.767807 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.767811 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.767823 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.767828 | controller | - ' exit 1' 2026-03-09 19:10:03.767832 | controller | - ' fi' 2026-03-09 19:10:03.767837 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.767841 | controller | - ' ' 2026-03-09 19:10:03.767845 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.767849 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.767854 | controller | | base64 -d)' 2026-03-09 19:10:03.767858 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.767862 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.767867 | controller | - ' exit 1' 2026-03-09 19:10:03.767871 | controller | - ' fi' 2026-03-09 19:10:03.767875 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.767881 | controller | - ' ' 2026-03-09 19:10:03.767886 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.767890 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.767894 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.767898 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.767903 | controller | --timeout=300s' 2026-03-09 19:10:03.767907 | controller | - ' ' 2026-03-09 19:10:03.767911 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.767915 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.767920 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.767926 | controller | - ' ' 2026-03-09 19:10:03.767930 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767934 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767938 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.767943 | controller | - ' ' 2026-03-09 19:10:03.767947 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.767958 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.767963 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.767967 | controller | available"' 2026-03-09 19:10:03.767971 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.767976 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.767980 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.767984 | controller | - ' exit 1' 2026-03-09 19:10:03.767988 | controller | - ' fi' 2026-03-09 19:10:03.767993 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.767997 | controller | - ' ' 2026-03-09 19:10:03.768001 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.768007 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.768013 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768027 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.768033 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768039 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.768045 | controller | -- \' 2026-03-09 19:10:03.768051 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768056 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.768063 | controller | - ' logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.768069 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.768073 | controller | - ' logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.768077 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.768082 | controller | found' 2026-03-09 19:10:03.768086 | controller | - ' logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.768090 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.768094 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.768099 | controller | - ' ' 2026-03-09 19:10:03.768103 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.768107 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.768111 | controller | - ' ' 2026-03-09 19:10:03.768115 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.768120 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.768133 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.768137 | controller | - ' exit 1' 2026-03-09 19:10:03.768142 | controller | - ' fi' 2026-03-09 19:10:03.768167 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.768177 | controller | - ' ' 2026-03-09 19:10:03.768182 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.768186 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.768190 | controller | | base64 -d)' 2026-03-09 19:10:03.768195 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.768199 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.768203 | controller | - ' exit 1' 2026-03-09 19:10:03.768207 | controller | - ' fi' 2026-03-09 19:10:03.768212 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.768216 | controller | - ' ' 2026-03-09 19:10:03.768220 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.768224 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.768229 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.768235 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.768240 | controller | --timeout=300s' 2026-03-09 19:10:03.768244 | controller | - ' ' 2026-03-09 19:10:03.768248 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.768252 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.768257 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.768261 | controller | - ' ' 2026-03-09 19:10:03.768265 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768269 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768274 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768278 | controller | - ' ' 2026-03-09 19:10:03.768282 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.768286 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.768291 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.768295 | controller | available"' 2026-03-09 19:10:03.768299 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.768303 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.768308 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.768312 | controller | - ' exit 1' 2026-03-09 19:10:03.768316 | controller | - ' fi' 2026-03-09 19:10:03.768320 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.768325 | controller | - ' ' 2026-03-09 19:10:03.768329 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.768333 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.768338 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768342 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.768346 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768350 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.768355 | controller | -- \' 2026-03-09 19:10:03.768359 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768365 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.768369 | controller | - ' logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.768374 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.768378 | controller | - ' logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.768382 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.768386 | controller | found' 2026-03-09 19:10:03.768391 | controller | - ' logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.768395 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.768399 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.768403 | controller | - ' ' 2026-03-09 19:10:03.768409 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.768414 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.768418 | controller | - ' ' 2026-03-09 19:10:03.768422 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.768427 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.768431 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.768435 | controller | - ' exit 1' 2026-03-09 19:10:03.768439 | controller | - ' fi' 2026-03-09 19:10:03.768452 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.768456 | controller | - ' ' 2026-03-09 19:10:03.768460 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.768465 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.768469 | controller | | base64 -d)' 2026-03-09 19:10:03.768473 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.768478 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.768482 | controller | - ' exit 1' 2026-03-09 19:10:03.768486 | controller | - ' fi' 2026-03-09 19:10:03.768490 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.768495 | controller | - ' ' 2026-03-09 19:10:03.768499 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.768503 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.768507 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.768512 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.768516 | controller | --timeout=300s' 2026-03-09 19:10:03.768520 | controller | - ' ' 2026-03-09 19:10:03.768524 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.768529 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.768533 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.768537 | controller | - ' ' 2026-03-09 19:10:03.768541 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768546 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768552 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768556 | controller | - ' ' 2026-03-09 19:10:03.768560 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.768567 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.768572 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.768576 | controller | available"' 2026-03-09 19:10:03.768580 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.768584 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.768589 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.768593 | controller | - ' exit 1' 2026-03-09 19:10:03.768597 | controller | - ' fi' 2026-03-09 19:10:03.768602 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.768606 | controller | - ' ' 2026-03-09 19:10:03.768610 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.768614 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.768619 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768623 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.768627 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768632 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.768636 | controller | -- \' 2026-03-09 19:10:03.768641 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768646 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.768652 | controller | - ' logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.768658 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.768664 | controller | - ' logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.768670 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.768675 | controller | found' 2026-03-09 19:10:03.768681 | controller | - ' logger.go:42: 18:57:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.768687 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.768692 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.768698 | controller | - ' ' 2026-03-09 19:10:03.768703 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.768709 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.768715 | controller | - ' ' 2026-03-09 19:10:03.768720 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.768726 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.768732 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.768738 | controller | - ' exit 1' 2026-03-09 19:10:03.768744 | controller | - ' fi' 2026-03-09 19:10:03.768749 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.768764 | controller | - ' ' 2026-03-09 19:10:03.768773 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.768779 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.768784 | controller | | base64 -d)' 2026-03-09 19:10:03.768790 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.768796 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.768802 | controller | - ' exit 1' 2026-03-09 19:10:03.768808 | controller | - ' fi' 2026-03-09 19:10:03.768818 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.768825 | controller | - ' ' 2026-03-09 19:10:03.768830 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.768836 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.768841 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.768847 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.768853 | controller | --timeout=300s' 2026-03-09 19:10:03.768859 | controller | - ' ' 2026-03-09 19:10:03.768864 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.768870 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.768876 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.768882 | controller | - ' ' 2026-03-09 19:10:03.768887 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768892 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768896 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.768900 | controller | - ' ' 2026-03-09 19:10:03.768908 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.768912 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.768916 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.768921 | controller | available"' 2026-03-09 19:10:03.768925 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.768929 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.768933 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.768937 | controller | - ' exit 1' 2026-03-09 19:10:03.768942 | controller | - ' fi' 2026-03-09 19:10:03.768946 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.768950 | controller | - ' ' 2026-03-09 19:10:03.768955 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.768959 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.768963 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768967 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.768971 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768976 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.768980 | controller | -- \' 2026-03-09 19:10:03.768984 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.768989 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.768993 | controller | - ' logger.go:42: 18:57:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.768997 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.769001 | controller | - ' logger.go:42: 18:57:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.769006 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.769015 | controller | found' 2026-03-09 19:10:03.769021 | controller | - ' logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.769027 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.769033 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.769039 | controller | - ' ' 2026-03-09 19:10:03.769045 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.769050 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.769054 | controller | - ' ' 2026-03-09 19:10:03.769058 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.769063 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.769067 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.769071 | controller | - ' exit 1' 2026-03-09 19:10:03.769076 | controller | - ' fi' 2026-03-09 19:10:03.769080 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.769084 | controller | - ' ' 2026-03-09 19:10:03.769098 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.769104 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.769108 | controller | | base64 -d)' 2026-03-09 19:10:03.769113 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.769119 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.769124 | controller | - ' exit 1' 2026-03-09 19:10:03.769128 | controller | - ' fi' 2026-03-09 19:10:03.769132 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.769137 | controller | - ' ' 2026-03-09 19:10:03.769141 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.769167 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.769175 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.769179 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.769183 | controller | --timeout=300s' 2026-03-09 19:10:03.769188 | controller | - ' ' 2026-03-09 19:10:03.769192 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.769196 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.769201 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.769205 | controller | - ' ' 2026-03-09 19:10:03.769209 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769213 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769218 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769222 | controller | - ' ' 2026-03-09 19:10:03.769226 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.769230 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.769235 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.769239 | controller | available"' 2026-03-09 19:10:03.769243 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.769247 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.769252 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.769258 | controller | - ' exit 1' 2026-03-09 19:10:03.769262 | controller | - ' fi' 2026-03-09 19:10:03.769267 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.769271 | controller | - ' ' 2026-03-09 19:10:03.769275 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.769279 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.769284 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769288 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.769292 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769296 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.769301 | controller | -- \' 2026-03-09 19:10:03.769305 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769309 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.769313 | controller | - ' logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.769318 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.769322 | controller | - ' logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.769326 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.769330 | controller | found' 2026-03-09 19:10:03.769335 | controller | - ' logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.769339 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.769343 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.769347 | controller | - ' ' 2026-03-09 19:10:03.769352 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.769356 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.769360 | controller | - ' ' 2026-03-09 19:10:03.769365 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.769369 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.769373 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.769377 | controller | - ' exit 1' 2026-03-09 19:10:03.769382 | controller | - ' fi' 2026-03-09 19:10:03.769386 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.769390 | controller | - ' ' 2026-03-09 19:10:03.769394 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.769406 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.769411 | controller | | base64 -d)' 2026-03-09 19:10:03.769415 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.769419 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.769424 | controller | - ' exit 1' 2026-03-09 19:10:03.769428 | controller | - ' fi' 2026-03-09 19:10:03.769432 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.769436 | controller | - ' ' 2026-03-09 19:10:03.769441 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.769445 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.769449 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.769453 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.769459 | controller | --timeout=300s' 2026-03-09 19:10:03.769464 | controller | - ' ' 2026-03-09 19:10:03.769468 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.769472 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.769477 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.769481 | controller | - ' ' 2026-03-09 19:10:03.769485 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769489 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769494 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769498 | controller | - ' ' 2026-03-09 19:10:03.769504 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.769508 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.769513 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.769519 | controller | available"' 2026-03-09 19:10:03.769523 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.769527 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.769532 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.769536 | controller | - ' exit 1' 2026-03-09 19:10:03.769540 | controller | - ' fi' 2026-03-09 19:10:03.769544 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.769549 | controller | - ' ' 2026-03-09 19:10:03.769553 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.769557 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.769561 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769566 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.769570 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769574 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.769578 | controller | -- \' 2026-03-09 19:10:03.769583 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769587 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.769593 | controller | - ' logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.769598 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.769602 | controller | - ' logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.769606 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.769611 | controller | found' 2026-03-09 19:10:03.769615 | controller | - ' logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.769619 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.769623 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.769628 | controller | - ' ' 2026-03-09 19:10:03.769632 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.769636 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.769642 | controller | - ' ' 2026-03-09 19:10:03.769646 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.769651 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.769655 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.769659 | controller | - ' exit 1' 2026-03-09 19:10:03.769663 | controller | - ' fi' 2026-03-09 19:10:03.769668 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.769672 | controller | - ' ' 2026-03-09 19:10:03.769676 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.769688 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.769692 | controller | | base64 -d)' 2026-03-09 19:10:03.769697 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.769701 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.769705 | controller | - ' exit 1' 2026-03-09 19:10:03.769710 | controller | - ' fi' 2026-03-09 19:10:03.769714 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.769718 | controller | - ' ' 2026-03-09 19:10:03.769723 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.769727 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.769731 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.769735 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.769740 | controller | --timeout=300s' 2026-03-09 19:10:03.769744 | controller | - ' ' 2026-03-09 19:10:03.769748 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.769752 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.769757 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.769761 | controller | - ' ' 2026-03-09 19:10:03.769765 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769770 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769774 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.769778 | controller | - ' ' 2026-03-09 19:10:03.769782 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.769787 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.769791 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.769795 | controller | available"' 2026-03-09 19:10:03.769799 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.769804 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.769808 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.769812 | controller | - ' exit 1' 2026-03-09 19:10:03.769817 | controller | - ' fi' 2026-03-09 19:10:03.769821 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.769825 | controller | - ' ' 2026-03-09 19:10:03.769829 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.769834 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.769838 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769844 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.769848 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769852 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.769857 | controller | -- \' 2026-03-09 19:10:03.769861 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.769865 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.769870 | controller | - ' logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.769874 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.769878 | controller | - ' logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.769882 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.769887 | controller | found' 2026-03-09 19:10:03.769891 | controller | - ' logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.769895 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.769900 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.769906 | controller | - ' ' 2026-03-09 19:10:03.769910 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.769915 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.769919 | controller | - ' ' 2026-03-09 19:10:03.769923 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.769927 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.769932 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.769936 | controller | - ' exit 1' 2026-03-09 19:10:03.769941 | controller | - ' fi' 2026-03-09 19:10:03.769945 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.769949 | controller | - ' ' 2026-03-09 19:10:03.769954 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.769958 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.769962 | controller | | base64 -d)' 2026-03-09 19:10:03.769973 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.769978 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.769982 | controller | - ' exit 1' 2026-03-09 19:10:03.769986 | controller | - ' fi' 2026-03-09 19:10:03.769991 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.769995 | controller | - ' ' 2026-03-09 19:10:03.769999 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.770004 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.770010 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.770016 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.770022 | controller | --timeout=300s' 2026-03-09 19:10:03.770028 | controller | - ' ' 2026-03-09 19:10:03.770034 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.770040 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.770044 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.770048 | controller | - ' ' 2026-03-09 19:10:03.770053 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770059 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770063 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770068 | controller | - ' ' 2026-03-09 19:10:03.770072 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.770076 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.770081 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.770085 | controller | available"' 2026-03-09 19:10:03.770089 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.770096 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.770100 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.770104 | controller | - ' exit 1' 2026-03-09 19:10:03.770109 | controller | - ' fi' 2026-03-09 19:10:03.770115 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.770119 | controller | - ' ' 2026-03-09 19:10:03.770123 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.770128 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.770132 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770136 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.770141 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770162 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.770169 | controller | -- \' 2026-03-09 19:10:03.770173 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770177 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.770182 | controller | - ' logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.770186 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.770191 | controller | - ' logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.770195 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.770199 | controller | found' 2026-03-09 19:10:03.770203 | controller | - ' logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.770208 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.770212 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.770216 | controller | - ' ' 2026-03-09 19:10:03.770221 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.770225 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.770229 | controller | - ' ' 2026-03-09 19:10:03.770233 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.770238 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.770242 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.770246 | controller | - ' exit 1' 2026-03-09 19:10:03.770251 | controller | - ' fi' 2026-03-09 19:10:03.770255 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.770259 | controller | - ' ' 2026-03-09 19:10:03.770264 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.770271 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.770276 | controller | | base64 -d)' 2026-03-09 19:10:03.770289 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.770294 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.770298 | controller | - ' exit 1' 2026-03-09 19:10:03.770303 | controller | - ' fi' 2026-03-09 19:10:03.770307 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.770311 | controller | - ' ' 2026-03-09 19:10:03.770316 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.770320 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.770324 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.770328 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.770333 | controller | --timeout=300s' 2026-03-09 19:10:03.770337 | controller | - ' ' 2026-03-09 19:10:03.770341 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.770345 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.770350 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.770354 | controller | - ' ' 2026-03-09 19:10:03.770358 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770362 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770367 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770371 | controller | - ' ' 2026-03-09 19:10:03.770375 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.770382 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.770386 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.770390 | controller | available"' 2026-03-09 19:10:03.770395 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.770399 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.770403 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.770407 | controller | - ' exit 1' 2026-03-09 19:10:03.770412 | controller | - ' fi' 2026-03-09 19:10:03.770416 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.770420 | controller | - ' ' 2026-03-09 19:10:03.770425 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.770429 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.770433 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770437 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.770442 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770446 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.770450 | controller | -- \' 2026-03-09 19:10:03.770454 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770460 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.770466 | controller | - ' logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.770471 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.770475 | controller | - ' logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.770479 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.770484 | controller | found' 2026-03-09 19:10:03.770488 | controller | - ' logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.770492 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.770496 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.770501 | controller | - ' ' 2026-03-09 19:10:03.770505 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.770509 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.770513 | controller | - ' ' 2026-03-09 19:10:03.770518 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.770522 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.770526 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.770530 | controller | - ' exit 1' 2026-03-09 19:10:03.770535 | controller | - ' fi' 2026-03-09 19:10:03.770539 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.770543 | controller | - ' ' 2026-03-09 19:10:03.770547 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.770552 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.770556 | controller | | base64 -d)' 2026-03-09 19:10:03.770560 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.770572 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.770577 | controller | - ' exit 1' 2026-03-09 19:10:03.770581 | controller | - ' fi' 2026-03-09 19:10:03.770587 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.770591 | controller | - ' ' 2026-03-09 19:10:03.770596 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.770600 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.770604 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.770608 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.770613 | controller | --timeout=300s' 2026-03-09 19:10:03.770619 | controller | - ' ' 2026-03-09 19:10:03.770623 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.770627 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.770632 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.770636 | controller | - ' ' 2026-03-09 19:10:03.770640 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770644 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770649 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770653 | controller | - ' ' 2026-03-09 19:10:03.770657 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.770661 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.770668 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.770672 | controller | available"' 2026-03-09 19:10:03.770676 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.770681 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.770685 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.770689 | controller | - ' exit 1' 2026-03-09 19:10:03.770693 | controller | - ' fi' 2026-03-09 19:10:03.770698 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.770702 | controller | - ' ' 2026-03-09 19:10:03.770706 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.770712 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.770717 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770721 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.770725 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770729 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.770734 | controller | -- \' 2026-03-09 19:10:03.770738 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770742 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.770746 | controller | - ' logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.770751 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.770757 | controller | - ' logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.770761 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.770765 | controller | found' 2026-03-09 19:10:03.770769 | controller | - ' logger.go:42: 18:57:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.770774 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.770778 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.770782 | controller | - ' ' 2026-03-09 19:10:03.770786 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.770791 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.770795 | controller | - ' ' 2026-03-09 19:10:03.770799 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.770804 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.770808 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.770812 | controller | - ' exit 1' 2026-03-09 19:10:03.770816 | controller | - ' fi' 2026-03-09 19:10:03.770821 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.770825 | controller | - ' ' 2026-03-09 19:10:03.770829 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.770833 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.770838 | controller | | base64 -d)' 2026-03-09 19:10:03.770842 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.770846 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.770857 | controller | - ' exit 1' 2026-03-09 19:10:03.770862 | controller | - ' fi' 2026-03-09 19:10:03.770866 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.770872 | controller | - ' ' 2026-03-09 19:10:03.770877 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.770881 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.770885 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.770889 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.770894 | controller | --timeout=300s' 2026-03-09 19:10:03.770898 | controller | - ' ' 2026-03-09 19:10:03.770902 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.770907 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.770911 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.770915 | controller | - ' ' 2026-03-09 19:10:03.770919 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770924 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770928 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.770932 | controller | - ' ' 2026-03-09 19:10:03.770937 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.770941 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.770945 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.770949 | controller | available"' 2026-03-09 19:10:03.770954 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.770958 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.770962 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.770966 | controller | - ' exit 1' 2026-03-09 19:10:03.770971 | controller | - ' fi' 2026-03-09 19:10:03.770975 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.770979 | controller | - ' ' 2026-03-09 19:10:03.770983 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.770988 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.770992 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.770996 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.771000 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771005 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.771011 | controller | -- \' 2026-03-09 19:10:03.771017 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771023 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.771030 | controller | - ' logger.go:42: 18:57:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.771036 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.771040 | controller | - ' logger.go:42: 18:57:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.771044 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.771049 | controller | found' 2026-03-09 19:10:03.771055 | controller | - ' logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.771059 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.771064 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.771068 | controller | - ' ' 2026-03-09 19:10:03.771072 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.771076 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.771081 | controller | - ' ' 2026-03-09 19:10:03.771085 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.771089 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.771094 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.771098 | controller | - ' exit 1' 2026-03-09 19:10:03.771102 | controller | - ' fi' 2026-03-09 19:10:03.771106 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.771111 | controller | - ' ' 2026-03-09 19:10:03.771115 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.771119 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.771124 | controller | | base64 -d)' 2026-03-09 19:10:03.771128 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.771132 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.771136 | controller | - ' exit 1' 2026-03-09 19:10:03.771141 | controller | - ' fi' 2026-03-09 19:10:03.771164 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.771172 | controller | - ' ' 2026-03-09 19:10:03.771176 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.771180 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.771185 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.771189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.771193 | controller | --timeout=300s' 2026-03-09 19:10:03.771197 | controller | - ' ' 2026-03-09 19:10:03.771202 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.771206 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.771210 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.771215 | controller | - ' ' 2026-03-09 19:10:03.771219 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771223 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771227 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771232 | controller | - ' ' 2026-03-09 19:10:03.771236 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.771240 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.771244 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.771249 | controller | available"' 2026-03-09 19:10:03.771253 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.771257 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.771261 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.771266 | controller | - ' exit 1' 2026-03-09 19:10:03.771270 | controller | - ' fi' 2026-03-09 19:10:03.771276 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.771280 | controller | - ' ' 2026-03-09 19:10:03.771285 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.771289 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.771293 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771299 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.771304 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771308 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.771312 | controller | -- \' 2026-03-09 19:10:03.771317 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771321 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.771325 | controller | - ' logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.771330 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.771334 | controller | - ' logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.771338 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.771342 | controller | found' 2026-03-09 19:10:03.771347 | controller | - ' logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.771351 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.771355 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.771360 | controller | - ' ' 2026-03-09 19:10:03.771364 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.771368 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.771372 | controller | - ' ' 2026-03-09 19:10:03.771376 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.771381 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.771385 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.771391 | controller | - ' exit 1' 2026-03-09 19:10:03.771397 | controller | - ' fi' 2026-03-09 19:10:03.771403 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.771409 | controller | - ' ' 2026-03-09 19:10:03.771417 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.771421 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.771426 | controller | | base64 -d)' 2026-03-09 19:10:03.771430 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.771434 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.771438 | controller | - ' exit 1' 2026-03-09 19:10:03.771443 | controller | - ' fi' 2026-03-09 19:10:03.771447 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.771459 | controller | - ' ' 2026-03-09 19:10:03.771464 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.771468 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.771472 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.771477 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.771481 | controller | --timeout=300s' 2026-03-09 19:10:03.771485 | controller | - ' ' 2026-03-09 19:10:03.771492 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.771496 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.771500 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.771505 | controller | - ' ' 2026-03-09 19:10:03.771509 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771513 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771517 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771522 | controller | - ' ' 2026-03-09 19:10:03.771526 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.771530 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.771534 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.771539 | controller | available"' 2026-03-09 19:10:03.771543 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.771547 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.771552 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.771556 | controller | - ' exit 1' 2026-03-09 19:10:03.771560 | controller | - ' fi' 2026-03-09 19:10:03.771564 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.771569 | controller | - ' ' 2026-03-09 19:10:03.771573 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.771577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.771582 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771586 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.771590 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771594 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.771599 | controller | -- \' 2026-03-09 19:10:03.771603 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771607 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.771612 | controller | - ' logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.771616 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.771620 | controller | - ' logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.771624 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.771629 | controller | found' 2026-03-09 19:10:03.771633 | controller | - ' logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.771637 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.771642 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.771646 | controller | - ' ' 2026-03-09 19:10:03.771650 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.771655 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.771704 | controller | - ' ' 2026-03-09 19:10:03.771708 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.771716 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.771720 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.771725 | controller | - ' exit 1' 2026-03-09 19:10:03.771731 | controller | - ' fi' 2026-03-09 19:10:03.771735 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.771739 | controller | - ' ' 2026-03-09 19:10:03.771744 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.771748 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.771752 | controller | | base64 -d)' 2026-03-09 19:10:03.771756 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.771761 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.771765 | controller | - ' exit 1' 2026-03-09 19:10:03.771769 | controller | - ' fi' 2026-03-09 19:10:03.771774 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.771778 | controller | - ' ' 2026-03-09 19:10:03.771790 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.771794 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.771799 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.771803 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.771807 | controller | --timeout=300s' 2026-03-09 19:10:03.771812 | controller | - ' ' 2026-03-09 19:10:03.771816 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.771820 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.771827 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.771831 | controller | - ' ' 2026-03-09 19:10:03.771841 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771845 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771849 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.771854 | controller | - ' ' 2026-03-09 19:10:03.771858 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.771862 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.771866 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.771871 | controller | available"' 2026-03-09 19:10:03.771875 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.771879 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.771883 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.771888 | controller | - ' exit 1' 2026-03-09 19:10:03.771892 | controller | - ' fi' 2026-03-09 19:10:03.771896 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.771900 | controller | - ' ' 2026-03-09 19:10:03.771905 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.771909 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.771913 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771917 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.771923 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771928 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.771932 | controller | -- \' 2026-03-09 19:10:03.771936 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.771949 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.771954 | controller | - ' logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.771958 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.771962 | controller | - ' logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.771966 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.771971 | controller | found' 2026-03-09 19:10:03.771975 | controller | - ' logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.771979 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.771983 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.771988 | controller | - ' ' 2026-03-09 19:10:03.771992 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.771996 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.772001 | controller | - ' ' 2026-03-09 19:10:03.772005 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.772011 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.772017 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.772023 | controller | - ' exit 1' 2026-03-09 19:10:03.772029 | controller | - ' fi' 2026-03-09 19:10:03.772035 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.772041 | controller | - ' ' 2026-03-09 19:10:03.772047 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.772052 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.772058 | controller | | base64 -d)' 2026-03-09 19:10:03.772064 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.772070 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.772075 | controller | - ' exit 1' 2026-03-09 19:10:03.772079 | controller | - ' fi' 2026-03-09 19:10:03.772083 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.772088 | controller | - ' ' 2026-03-09 19:10:03.772092 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.772105 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.772110 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.772115 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.772121 | controller | --timeout=300s' 2026-03-09 19:10:03.772125 | controller | - ' ' 2026-03-09 19:10:03.772130 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.772135 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.772140 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.772164 | controller | - ' ' 2026-03-09 19:10:03.772171 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772178 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772183 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772187 | controller | - ' ' 2026-03-09 19:10:03.772191 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.772196 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.772200 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.772204 | controller | available"' 2026-03-09 19:10:03.772208 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.772212 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.772217 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.772221 | controller | - ' exit 1' 2026-03-09 19:10:03.772225 | controller | - ' fi' 2026-03-09 19:10:03.772230 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.772234 | controller | - ' ' 2026-03-09 19:10:03.772238 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.772242 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.772246 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772251 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.772255 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772259 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.772264 | controller | -- \' 2026-03-09 19:10:03.772268 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772272 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.772276 | controller | - ' logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.772281 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.772285 | controller | - ' logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.772289 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.772293 | controller | found' 2026-03-09 19:10:03.772298 | controller | - ' logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.772302 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.772306 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.772310 | controller | - ' ' 2026-03-09 19:10:03.772315 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.772319 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.772323 | controller | - ' ' 2026-03-09 19:10:03.772327 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.772332 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.772336 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.772340 | controller | - ' exit 1' 2026-03-09 19:10:03.772344 | controller | - ' fi' 2026-03-09 19:10:03.772349 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.772353 | controller | - ' ' 2026-03-09 19:10:03.772357 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.772363 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.772367 | controller | | base64 -d)' 2026-03-09 19:10:03.772372 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.772376 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.772380 | controller | - ' exit 1' 2026-03-09 19:10:03.772384 | controller | - ' fi' 2026-03-09 19:10:03.772389 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.772393 | controller | - ' ' 2026-03-09 19:10:03.772397 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.772409 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.772414 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.772418 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.772422 | controller | --timeout=300s' 2026-03-09 19:10:03.772427 | controller | - ' ' 2026-03-09 19:10:03.772431 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.772435 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.772440 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.772444 | controller | - ' ' 2026-03-09 19:10:03.772448 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772452 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772457 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772461 | controller | - ' ' 2026-03-09 19:10:03.772465 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.772469 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.772474 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.772478 | controller | available"' 2026-03-09 19:10:03.772482 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.772486 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.772491 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.772495 | controller | - ' exit 1' 2026-03-09 19:10:03.772499 | controller | - ' fi' 2026-03-09 19:10:03.772503 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.772508 | controller | - ' ' 2026-03-09 19:10:03.772512 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.772516 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.772520 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772525 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.772529 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772533 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.772538 | controller | -- \' 2026-03-09 19:10:03.772542 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772546 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.772553 | controller | - ' logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.772557 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.772563 | controller | - ' logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.772568 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.772572 | controller | found' 2026-03-09 19:10:03.772576 | controller | - ' logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.772581 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.772585 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.772589 | controller | - ' ' 2026-03-09 19:10:03.772593 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.772598 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.772602 | controller | - ' ' 2026-03-09 19:10:03.772606 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.772610 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.772615 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.772619 | controller | - ' exit 1' 2026-03-09 19:10:03.772623 | controller | - ' fi' 2026-03-09 19:10:03.772627 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.772632 | controller | - ' ' 2026-03-09 19:10:03.772636 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.772640 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.772645 | controller | | base64 -d)' 2026-03-09 19:10:03.772649 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.772653 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.772657 | controller | - ' exit 1' 2026-03-09 19:10:03.772662 | controller | - ' fi' 2026-03-09 19:10:03.772666 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.772670 | controller | - ' ' 2026-03-09 19:10:03.772674 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.772679 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.772690 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.772694 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.772699 | controller | --timeout=300s' 2026-03-09 19:10:03.772703 | controller | - ' ' 2026-03-09 19:10:03.772707 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.772712 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.772716 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.772720 | controller | - ' ' 2026-03-09 19:10:03.772724 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772729 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772733 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.772737 | controller | - ' ' 2026-03-09 19:10:03.772741 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.772746 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.772752 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.772756 | controller | available"' 2026-03-09 19:10:03.772760 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.772766 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.772771 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.772775 | controller | - ' exit 1' 2026-03-09 19:10:03.772781 | controller | - ' fi' 2026-03-09 19:10:03.772785 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.772789 | controller | - ' ' 2026-03-09 19:10:03.772794 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.772798 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.772802 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772806 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.772812 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772817 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.772822 | controller | -- \' 2026-03-09 19:10:03.772827 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.772831 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.772835 | controller | - ' logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.772839 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.772844 | controller | - ' logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.772848 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.772852 | controller | found' 2026-03-09 19:10:03.772856 | controller | - ' logger.go:42: 18:57:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.772861 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.772865 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.772869 | controller | - ' ' 2026-03-09 19:10:03.772874 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.772878 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.772882 | controller | - ' ' 2026-03-09 19:10:03.772886 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.772890 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.772895 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.772899 | controller | - ' exit 1' 2026-03-09 19:10:03.772903 | controller | - ' fi' 2026-03-09 19:10:03.772908 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.772912 | controller | - ' ' 2026-03-09 19:10:03.772916 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.772920 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.772925 | controller | | base64 -d)' 2026-03-09 19:10:03.772929 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.772933 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.772937 | controller | - ' exit 1' 2026-03-09 19:10:03.772942 | controller | - ' fi' 2026-03-09 19:10:03.772946 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.772950 | controller | - ' ' 2026-03-09 19:10:03.772958 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.772962 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.772966 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.772978 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.772982 | controller | --timeout=300s' 2026-03-09 19:10:03.772987 | controller | - ' ' 2026-03-09 19:10:03.772991 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.772995 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.773000 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.773004 | controller | - ' ' 2026-03-09 19:10:03.773010 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773016 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773022 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773028 | controller | - ' ' 2026-03-09 19:10:03.773034 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.773040 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.773044 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.773048 | controller | available"' 2026-03-09 19:10:03.773053 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.773057 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.773061 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.773065 | controller | - ' exit 1' 2026-03-09 19:10:03.773070 | controller | - ' fi' 2026-03-09 19:10:03.773074 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.773078 | controller | - ' ' 2026-03-09 19:10:03.773083 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.773087 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.773091 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773095 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.773100 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773104 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.773108 | controller | -- \' 2026-03-09 19:10:03.773112 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773117 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.773121 | controller | - ' logger.go:42: 18:57:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.773125 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.773129 | controller | - ' logger.go:42: 18:57:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.773136 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.773141 | controller | found' 2026-03-09 19:10:03.773158 | controller | - ' logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.773167 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.773172 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.773176 | controller | - ' ' 2026-03-09 19:10:03.773180 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.773185 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.773189 | controller | - ' ' 2026-03-09 19:10:03.773193 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.773197 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.773202 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.773206 | controller | - ' exit 1' 2026-03-09 19:10:03.773210 | controller | - ' fi' 2026-03-09 19:10:03.773215 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.773219 | controller | - ' ' 2026-03-09 19:10:03.773223 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.773227 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.773232 | controller | | base64 -d)' 2026-03-09 19:10:03.773236 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.773242 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.773246 | controller | - ' exit 1' 2026-03-09 19:10:03.773251 | controller | - ' fi' 2026-03-09 19:10:03.773255 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.773259 | controller | - ' ' 2026-03-09 19:10:03.773263 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.773268 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.773272 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.773285 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.773290 | controller | --timeout=300s' 2026-03-09 19:10:03.773294 | controller | - ' ' 2026-03-09 19:10:03.773298 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.773303 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.773307 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.773311 | controller | - ' ' 2026-03-09 19:10:03.773316 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773320 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773324 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773328 | controller | - ' ' 2026-03-09 19:10:03.773333 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.773337 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.773341 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.773345 | controller | available"' 2026-03-09 19:10:03.773350 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.773354 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.773358 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.773362 | controller | - ' exit 1' 2026-03-09 19:10:03.773367 | controller | - ' fi' 2026-03-09 19:10:03.773371 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.773377 | controller | - ' ' 2026-03-09 19:10:03.773381 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.773386 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.773390 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773394 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.773399 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773403 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.773407 | controller | -- \' 2026-03-09 19:10:03.773411 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773416 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.773420 | controller | - ' logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.773424 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.773428 | controller | - ' logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.773433 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.773437 | controller | found' 2026-03-09 19:10:03.773441 | controller | - ' logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.773445 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.773450 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.773454 | controller | - ' ' 2026-03-09 19:10:03.773458 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.773463 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.773467 | controller | - ' ' 2026-03-09 19:10:03.773471 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.773475 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.773480 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.773484 | controller | - ' exit 1' 2026-03-09 19:10:03.773488 | controller | - ' fi' 2026-03-09 19:10:03.773493 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.773497 | controller | - ' ' 2026-03-09 19:10:03.773501 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.773506 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.773510 | controller | | base64 -d)' 2026-03-09 19:10:03.773514 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.773518 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.773523 | controller | - ' exit 1' 2026-03-09 19:10:03.773527 | controller | - ' fi' 2026-03-09 19:10:03.773531 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.773535 | controller | - ' ' 2026-03-09 19:10:03.773540 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.773544 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.773548 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.773552 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.773557 | controller | --timeout=300s' 2026-03-09 19:10:03.773567 | controller | - ' ' 2026-03-09 19:10:03.773572 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.773578 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.773582 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.773587 | controller | - ' ' 2026-03-09 19:10:03.773591 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773595 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773599 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773604 | controller | - ' ' 2026-03-09 19:10:03.773608 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.773612 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.773616 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.773621 | controller | available"' 2026-03-09 19:10:03.773625 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.773629 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.773634 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.773638 | controller | - ' exit 1' 2026-03-09 19:10:03.773642 | controller | - ' fi' 2026-03-09 19:10:03.773646 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.773651 | controller | - ' ' 2026-03-09 19:10:03.773655 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.773659 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.773664 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.773672 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773676 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.773681 | controller | -- \' 2026-03-09 19:10:03.773685 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773689 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.773693 | controller | - ' logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.773698 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.773704 | controller | - ' logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.773708 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.773713 | controller | found' 2026-03-09 19:10:03.773717 | controller | - ' logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.773723 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.773727 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.773732 | controller | - ' ' 2026-03-09 19:10:03.773736 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.773741 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.773745 | controller | - ' ' 2026-03-09 19:10:03.773749 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.773755 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.773759 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.773764 | controller | - ' exit 1' 2026-03-09 19:10:03.773768 | controller | - ' fi' 2026-03-09 19:10:03.773772 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.773776 | controller | - ' ' 2026-03-09 19:10:03.773781 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.773785 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.773789 | controller | | base64 -d)' 2026-03-09 19:10:03.773793 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.773798 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.773802 | controller | - ' exit 1' 2026-03-09 19:10:03.773806 | controller | - ' fi' 2026-03-09 19:10:03.773810 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.773815 | controller | - ' ' 2026-03-09 19:10:03.773819 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.773823 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.773827 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.773832 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.773836 | controller | --timeout=300s' 2026-03-09 19:10:03.773842 | controller | - ' ' 2026-03-09 19:10:03.773854 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.773859 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.773863 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.773867 | controller | - ' ' 2026-03-09 19:10:03.773872 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773878 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773882 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.773886 | controller | - ' ' 2026-03-09 19:10:03.773890 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.773895 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.773899 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.773903 | controller | available"' 2026-03-09 19:10:03.773908 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.773912 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.773916 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.773920 | controller | - ' exit 1' 2026-03-09 19:10:03.773925 | controller | - ' fi' 2026-03-09 19:10:03.773929 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.773933 | controller | - ' ' 2026-03-09 19:10:03.773937 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.773942 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.773946 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773950 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.773956 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773960 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.773965 | controller | -- \' 2026-03-09 19:10:03.773969 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.773973 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.773977 | controller | - ' logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.773982 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.773986 | controller | - ' logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.773990 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.773994 | controller | found' 2026-03-09 19:10:03.773999 | controller | - ' logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.774003 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.774009 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.774015 | controller | - ' ' 2026-03-09 19:10:03.774021 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.774027 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.774033 | controller | - ' ' 2026-03-09 19:10:03.774038 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.774042 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.774047 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.774051 | controller | - ' exit 1' 2026-03-09 19:10:03.774055 | controller | - ' fi' 2026-03-09 19:10:03.774059 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.774064 | controller | - ' ' 2026-03-09 19:10:03.774070 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.774075 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.774079 | controller | | base64 -d)' 2026-03-09 19:10:03.774083 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.774087 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.774092 | controller | - ' exit 1' 2026-03-09 19:10:03.774096 | controller | - ' fi' 2026-03-09 19:10:03.774100 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.774104 | controller | - ' ' 2026-03-09 19:10:03.774109 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.774113 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.774117 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.774121 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.774126 | controller | --timeout=300s' 2026-03-09 19:10:03.774130 | controller | - ' ' 2026-03-09 19:10:03.774142 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.774163 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.774167 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.774171 | controller | - ' ' 2026-03-09 19:10:03.774176 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774180 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774186 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774191 | controller | - ' ' 2026-03-09 19:10:03.774195 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.774199 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.774204 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.774208 | controller | available"' 2026-03-09 19:10:03.774212 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.774216 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.774221 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.774225 | controller | - ' exit 1' 2026-03-09 19:10:03.774229 | controller | - ' fi' 2026-03-09 19:10:03.774233 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.774238 | controller | - ' ' 2026-03-09 19:10:03.774242 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.774246 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.774250 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774255 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.774259 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774263 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.774267 | controller | -- \' 2026-03-09 19:10:03.774272 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774286 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.774291 | controller | - ' logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.774295 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.774299 | controller | - ' logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.774303 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.774308 | controller | found' 2026-03-09 19:10:03.774312 | controller | - ' logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.774317 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.774321 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.774326 | controller | - ' ' 2026-03-09 19:10:03.774332 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.774337 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.774341 | controller | - ' ' 2026-03-09 19:10:03.774345 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.774349 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.774354 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.774358 | controller | - ' exit 1' 2026-03-09 19:10:03.774362 | controller | - ' fi' 2026-03-09 19:10:03.774366 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.774371 | controller | - ' ' 2026-03-09 19:10:03.774375 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.774379 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.774386 | controller | | base64 -d)' 2026-03-09 19:10:03.774391 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.774395 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.774399 | controller | - ' exit 1' 2026-03-09 19:10:03.774403 | controller | - ' fi' 2026-03-09 19:10:03.774408 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.774412 | controller | - ' ' 2026-03-09 19:10:03.774416 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.774420 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.774425 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.774429 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.774433 | controller | --timeout=300s' 2026-03-09 19:10:03.774437 | controller | - ' ' 2026-03-09 19:10:03.774442 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.774453 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.774458 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.774462 | controller | - ' ' 2026-03-09 19:10:03.774466 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774471 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774475 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774479 | controller | - ' ' 2026-03-09 19:10:03.774483 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.774488 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.774492 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.774496 | controller | available"' 2026-03-09 19:10:03.774500 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.774504 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.774509 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.774513 | controller | - ' exit 1' 2026-03-09 19:10:03.774517 | controller | - ' fi' 2026-03-09 19:10:03.774522 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.774526 | controller | - ' ' 2026-03-09 19:10:03.774530 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.774534 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.774539 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774543 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.774547 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774551 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.774555 | controller | -- \' 2026-03-09 19:10:03.774560 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774564 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.774570 | controller | - ' logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.774576 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.774580 | controller | - ' logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.774585 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.774589 | controller | found' 2026-03-09 19:10:03.774593 | controller | - ' logger.go:42: 18:57:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.774597 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.774602 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.774606 | controller | - ' ' 2026-03-09 19:10:03.774610 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.774614 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.774619 | controller | - ' ' 2026-03-09 19:10:03.774623 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.774627 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.774631 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.774636 | controller | - ' exit 1' 2026-03-09 19:10:03.774640 | controller | - ' fi' 2026-03-09 19:10:03.774644 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.774648 | controller | - ' ' 2026-03-09 19:10:03.774653 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.774657 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.774661 | controller | | base64 -d)' 2026-03-09 19:10:03.774665 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.774670 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.774674 | controller | - ' exit 1' 2026-03-09 19:10:03.774678 | controller | - ' fi' 2026-03-09 19:10:03.774683 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.774687 | controller | - ' ' 2026-03-09 19:10:03.774691 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.774695 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.774700 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.774704 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.774708 | controller | --timeout=300s' 2026-03-09 19:10:03.774714 | controller | - ' ' 2026-03-09 19:10:03.774718 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.774729 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.774734 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.774738 | controller | - ' ' 2026-03-09 19:10:03.774742 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774747 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774751 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.774755 | controller | - ' ' 2026-03-09 19:10:03.774760 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.774764 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.774768 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.774774 | controller | available"' 2026-03-09 19:10:03.774778 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.774783 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.774787 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.774791 | controller | - ' exit 1' 2026-03-09 19:10:03.774795 | controller | - ' fi' 2026-03-09 19:10:03.774800 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.774804 | controller | - ' ' 2026-03-09 19:10:03.774808 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.774812 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.774817 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774821 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.774825 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774829 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.774834 | controller | -- \' 2026-03-09 19:10:03.774838 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.774842 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.774847 | controller | - ' logger.go:42: 18:57:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.774851 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.774857 | controller | - ' logger.go:42: 18:57:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.774861 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.774865 | controller | found' 2026-03-09 19:10:03.774870 | controller | - ' logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.774874 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.774878 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.774883 | controller | - ' ' 2026-03-09 19:10:03.774887 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.774891 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.774895 | controller | - ' ' 2026-03-09 19:10:03.774900 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.774904 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.774908 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.774912 | controller | - ' exit 1' 2026-03-09 19:10:03.774918 | controller | - ' fi' 2026-03-09 19:10:03.774922 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.774927 | controller | - ' ' 2026-03-09 19:10:03.774932 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.774937 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.774941 | controller | | base64 -d)' 2026-03-09 19:10:03.774945 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.774950 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.774954 | controller | - ' exit 1' 2026-03-09 19:10:03.774958 | controller | - ' fi' 2026-03-09 19:10:03.774962 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.774967 | controller | - ' ' 2026-03-09 19:10:03.774971 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.774977 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.774981 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.774985 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.774990 | controller | --timeout=300s' 2026-03-09 19:10:03.774994 | controller | - ' ' 2026-03-09 19:10:03.774998 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.775003 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.775016 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.775024 | controller | - ' ' 2026-03-09 19:10:03.775030 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775036 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775040 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775045 | controller | - ' ' 2026-03-09 19:10:03.775049 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.775053 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.775058 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.775062 | controller | available"' 2026-03-09 19:10:03.775066 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.775070 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.775074 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.775079 | controller | - ' exit 1' 2026-03-09 19:10:03.775083 | controller | - ' fi' 2026-03-09 19:10:03.775087 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.775091 | controller | - ' ' 2026-03-09 19:10:03.775096 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.775100 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.775105 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775110 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.775115 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775119 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.775125 | controller | -- \' 2026-03-09 19:10:03.775130 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775135 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.775139 | controller | - ' logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.775143 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.775166 | controller | - ' logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.775170 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.775175 | controller | found' 2026-03-09 19:10:03.775179 | controller | - ' logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.775183 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.775190 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.775194 | controller | - ' ' 2026-03-09 19:10:03.775198 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.775203 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.775207 | controller | - ' ' 2026-03-09 19:10:03.775211 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.775218 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.775222 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.775226 | controller | - ' exit 1' 2026-03-09 19:10:03.775231 | controller | - ' fi' 2026-03-09 19:10:03.775235 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.775239 | controller | - ' ' 2026-03-09 19:10:03.775243 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.775247 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.775252 | controller | | base64 -d)' 2026-03-09 19:10:03.775256 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.775260 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.775265 | controller | - ' exit 1' 2026-03-09 19:10:03.775269 | controller | - ' fi' 2026-03-09 19:10:03.775273 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.775277 | controller | - ' ' 2026-03-09 19:10:03.775282 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.775286 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.775290 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.775294 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.775298 | controller | --timeout=300s' 2026-03-09 19:10:03.775303 | controller | - ' ' 2026-03-09 19:10:03.775307 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.775311 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.775324 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.775328 | controller | - ' ' 2026-03-09 19:10:03.775332 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775337 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775341 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775346 | controller | - ' ' 2026-03-09 19:10:03.775350 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.775354 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.775358 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.775363 | controller | available"' 2026-03-09 19:10:03.775367 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.775373 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.775377 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.775381 | controller | - ' exit 1' 2026-03-09 19:10:03.775386 | controller | - ' fi' 2026-03-09 19:10:03.775390 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.775396 | controller | - ' ' 2026-03-09 19:10:03.775400 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.775404 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.775409 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775413 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.775417 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775421 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.775426 | controller | -- \' 2026-03-09 19:10:03.775430 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775434 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.775439 | controller | - ' logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.775443 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.775447 | controller | - ' logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.775451 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.775456 | controller | found' 2026-03-09 19:10:03.775460 | controller | - ' logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.775464 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.775468 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.775473 | controller | - ' ' 2026-03-09 19:10:03.775477 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.775481 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.775486 | controller | - ' ' 2026-03-09 19:10:03.775491 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.775495 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.775500 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.775504 | controller | - ' exit 1' 2026-03-09 19:10:03.775508 | controller | - ' fi' 2026-03-09 19:10:03.775513 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.775517 | controller | - ' ' 2026-03-09 19:10:03.775521 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.775525 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.775530 | controller | | base64 -d)' 2026-03-09 19:10:03.775534 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.775538 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.775542 | controller | - ' exit 1' 2026-03-09 19:10:03.775547 | controller | - ' fi' 2026-03-09 19:10:03.775551 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.775555 | controller | - ' ' 2026-03-09 19:10:03.775560 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.775564 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.775568 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.775572 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.775577 | controller | --timeout=300s' 2026-03-09 19:10:03.775581 | controller | - ' ' 2026-03-09 19:10:03.775585 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.775591 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.775595 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.775606 | controller | - ' ' 2026-03-09 19:10:03.775611 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775615 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775619 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775624 | controller | - ' ' 2026-03-09 19:10:03.775628 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.775632 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.775637 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.775641 | controller | available"' 2026-03-09 19:10:03.775645 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.775649 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.775672 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.775676 | controller | - ' exit 1' 2026-03-09 19:10:03.775681 | controller | - ' fi' 2026-03-09 19:10:03.775685 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.775689 | controller | - ' ' 2026-03-09 19:10:03.775693 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.775698 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.775702 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775706 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.775710 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775715 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.775719 | controller | -- \' 2026-03-09 19:10:03.775723 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775727 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.775732 | controller | - ' logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.775736 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.775740 | controller | - ' logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.775744 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.775749 | controller | found' 2026-03-09 19:10:03.775753 | controller | - ' logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.775757 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.775762 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.775766 | controller | - ' ' 2026-03-09 19:10:03.775770 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.775774 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.775779 | controller | - ' ' 2026-03-09 19:10:03.775783 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.775787 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.775795 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.775799 | controller | - ' exit 1' 2026-03-09 19:10:03.775803 | controller | - ' fi' 2026-03-09 19:10:03.775808 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.775812 | controller | - ' ' 2026-03-09 19:10:03.775816 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.775820 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.775825 | controller | | base64 -d)' 2026-03-09 19:10:03.775829 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.775833 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.775837 | controller | - ' exit 1' 2026-03-09 19:10:03.775842 | controller | - ' fi' 2026-03-09 19:10:03.775846 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.775850 | controller | - ' ' 2026-03-09 19:10:03.775855 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.775859 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.775863 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.775867 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.775872 | controller | --timeout=300s' 2026-03-09 19:10:03.775876 | controller | - ' ' 2026-03-09 19:10:03.775880 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.775884 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.775889 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.775893 | controller | - ' ' 2026-03-09 19:10:03.775906 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775911 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775916 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.775921 | controller | - ' ' 2026-03-09 19:10:03.775925 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.775929 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.775934 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.775938 | controller | available"' 2026-03-09 19:10:03.775942 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.775947 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.775951 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.775955 | controller | - ' exit 1' 2026-03-09 19:10:03.775959 | controller | - ' fi' 2026-03-09 19:10:03.775964 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.775968 | controller | - ' ' 2026-03-09 19:10:03.775974 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.775979 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.775983 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775987 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.775992 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.775998 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.776002 | controller | -- \' 2026-03-09 19:10:03.776008 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776014 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.776020 | controller | - ' logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.776026 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.776032 | controller | - ' logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.776038 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.776044 | controller | found' 2026-03-09 19:10:03.776049 | controller | - ' logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.776053 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.776057 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.776062 | controller | - ' ' 2026-03-09 19:10:03.776066 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.776070 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.776074 | controller | - ' ' 2026-03-09 19:10:03.776079 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.776083 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.776087 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.776092 | controller | - ' exit 1' 2026-03-09 19:10:03.776096 | controller | - ' fi' 2026-03-09 19:10:03.776100 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.776107 | controller | - ' ' 2026-03-09 19:10:03.776111 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.776116 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.776120 | controller | | base64 -d)' 2026-03-09 19:10:03.776124 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.776128 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.776133 | controller | - ' exit 1' 2026-03-09 19:10:03.776137 | controller | - ' fi' 2026-03-09 19:10:03.776141 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.776161 | controller | - ' ' 2026-03-09 19:10:03.776165 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.776170 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.776174 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.776178 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.776182 | controller | --timeout=300s' 2026-03-09 19:10:03.776187 | controller | - ' ' 2026-03-09 19:10:03.776191 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.776195 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.776199 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.776204 | controller | - ' ' 2026-03-09 19:10:03.776216 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776221 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776227 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776232 | controller | - ' ' 2026-03-09 19:10:03.776236 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.776240 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.776245 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.776249 | controller | available"' 2026-03-09 19:10:03.776253 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.776257 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.776262 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.776266 | controller | - ' exit 1' 2026-03-09 19:10:03.776270 | controller | - ' fi' 2026-03-09 19:10:03.776275 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.776279 | controller | - ' ' 2026-03-09 19:10:03.776283 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.776287 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.776292 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776296 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.776300 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776304 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.776309 | controller | -- \' 2026-03-09 19:10:03.776313 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776317 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.776321 | controller | - ' logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.776326 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.776330 | controller | - ' logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.776334 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.776338 | controller | found' 2026-03-09 19:10:03.776343 | controller | - ' logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.776347 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.776351 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.776355 | controller | - ' ' 2026-03-09 19:10:03.776360 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.776364 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.776368 | controller | - ' ' 2026-03-09 19:10:03.776372 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.776377 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.776381 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.776385 | controller | - ' exit 1' 2026-03-09 19:10:03.776389 | controller | - ' fi' 2026-03-09 19:10:03.776394 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.776398 | controller | - ' ' 2026-03-09 19:10:03.776402 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.776406 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.776411 | controller | | base64 -d)' 2026-03-09 19:10:03.776416 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.776421 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.776425 | controller | - ' exit 1' 2026-03-09 19:10:03.776429 | controller | - ' fi' 2026-03-09 19:10:03.776434 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.776438 | controller | - ' ' 2026-03-09 19:10:03.776442 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.776446 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.776451 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.776455 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.776459 | controller | --timeout=300s' 2026-03-09 19:10:03.776463 | controller | - ' ' 2026-03-09 19:10:03.776468 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.776472 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.776476 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.776480 | controller | - ' ' 2026-03-09 19:10:03.776485 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776496 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776500 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776504 | controller | - ' ' 2026-03-09 19:10:03.776509 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.776513 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.776517 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.776521 | controller | available"' 2026-03-09 19:10:03.776526 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.776530 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.776534 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.776538 | controller | - ' exit 1' 2026-03-09 19:10:03.776543 | controller | - ' fi' 2026-03-09 19:10:03.776547 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.776551 | controller | - ' ' 2026-03-09 19:10:03.776556 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.776560 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.776564 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776568 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.776573 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.776581 | controller | -- \' 2026-03-09 19:10:03.776586 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776590 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.776594 | controller | - ' logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.776600 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.776604 | controller | - ' logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.776609 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.776613 | controller | found' 2026-03-09 19:10:03.776617 | controller | - ' logger.go:42: 18:57:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.776621 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.776626 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.776630 | controller | - ' ' 2026-03-09 19:10:03.776634 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.776638 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.776643 | controller | - ' ' 2026-03-09 19:10:03.776647 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.776651 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.776655 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.776660 | controller | - ' exit 1' 2026-03-09 19:10:03.776664 | controller | - ' fi' 2026-03-09 19:10:03.776668 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.776673 | controller | - ' ' 2026-03-09 19:10:03.776679 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.776683 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.776687 | controller | | base64 -d)' 2026-03-09 19:10:03.776692 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.776696 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.776700 | controller | - ' exit 1' 2026-03-09 19:10:03.776705 | controller | - ' fi' 2026-03-09 19:10:03.776709 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.776713 | controller | - ' ' 2026-03-09 19:10:03.776717 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.776722 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.776727 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.776732 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.776736 | controller | --timeout=300s' 2026-03-09 19:10:03.776740 | controller | - ' ' 2026-03-09 19:10:03.776744 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.776749 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.776753 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.776757 | controller | - ' ' 2026-03-09 19:10:03.776762 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776773 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776778 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.776782 | controller | - ' ' 2026-03-09 19:10:03.776786 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.776791 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.776795 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.776799 | controller | available"' 2026-03-09 19:10:03.776805 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.776810 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.776814 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.776818 | controller | - ' exit 1' 2026-03-09 19:10:03.776822 | controller | - ' fi' 2026-03-09 19:10:03.776827 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.776831 | controller | - ' ' 2026-03-09 19:10:03.776835 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.776839 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.776844 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776848 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.776852 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776857 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.776861 | controller | -- \' 2026-03-09 19:10:03.776865 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.776870 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.776876 | controller | - ' logger.go:42: 18:57:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.776880 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.776884 | controller | - ' logger.go:42: 18:57:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.776888 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.776893 | controller | found' 2026-03-09 19:10:03.776897 | controller | - ' logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.776901 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.776905 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.776910 | controller | - ' ' 2026-03-09 19:10:03.776914 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.776918 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.776922 | controller | - ' ' 2026-03-09 19:10:03.776927 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.776931 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.776935 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.776940 | controller | - ' exit 1' 2026-03-09 19:10:03.776944 | controller | - ' fi' 2026-03-09 19:10:03.776948 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.776952 | controller | - ' ' 2026-03-09 19:10:03.776957 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.776961 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.776965 | controller | | base64 -d)' 2026-03-09 19:10:03.776969 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.776974 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.776978 | controller | - ' exit 1' 2026-03-09 19:10:03.776982 | controller | - ' fi' 2026-03-09 19:10:03.776986 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.776991 | controller | - ' ' 2026-03-09 19:10:03.776995 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.776999 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.777005 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.777009 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.777014 | controller | --timeout=300s' 2026-03-09 19:10:03.777018 | controller | - ' ' 2026-03-09 19:10:03.777022 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.777027 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.777031 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.777035 | controller | - ' ' 2026-03-09 19:10:03.777041 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777045 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777057 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777062 | controller | - ' ' 2026-03-09 19:10:03.777066 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.777070 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.777075 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.777079 | controller | available"' 2026-03-09 19:10:03.777083 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.777088 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.777092 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.777096 | controller | - ' exit 1' 2026-03-09 19:10:03.777100 | controller | - ' fi' 2026-03-09 19:10:03.777104 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.777109 | controller | - ' ' 2026-03-09 19:10:03.777113 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.777117 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.777121 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777126 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.777130 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777134 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.777138 | controller | -- \' 2026-03-09 19:10:03.777143 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777160 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.777164 | controller | - ' logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.777168 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.777173 | controller | - ' logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.777177 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.777181 | controller | found' 2026-03-09 19:10:03.777185 | controller | - ' logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.777190 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.777194 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.777203 | controller | - ' ' 2026-03-09 19:10:03.777208 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.777212 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.777217 | controller | - ' ' 2026-03-09 19:10:03.777221 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.777225 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.777230 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.777234 | controller | - ' exit 1' 2026-03-09 19:10:03.777239 | controller | - ' fi' 2026-03-09 19:10:03.777243 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.777247 | controller | - ' ' 2026-03-09 19:10:03.777252 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.777256 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.777260 | controller | | base64 -d)' 2026-03-09 19:10:03.777265 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.777270 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.777275 | controller | - ' exit 1' 2026-03-09 19:10:03.777279 | controller | - ' fi' 2026-03-09 19:10:03.777283 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.777288 | controller | - ' ' 2026-03-09 19:10:03.777292 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.777296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.777301 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.777306 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.777311 | controller | --timeout=300s' 2026-03-09 19:10:03.777315 | controller | - ' ' 2026-03-09 19:10:03.777319 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.777324 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.777328 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.777332 | controller | - ' ' 2026-03-09 19:10:03.777337 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777341 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777353 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777357 | controller | - ' ' 2026-03-09 19:10:03.777362 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.777366 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.777370 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.777375 | controller | available"' 2026-03-09 19:10:03.777379 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.777383 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.777388 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.777392 | controller | - ' exit 1' 2026-03-09 19:10:03.777396 | controller | - ' fi' 2026-03-09 19:10:03.777400 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.777405 | controller | - ' ' 2026-03-09 19:10:03.777409 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.777415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.777420 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777424 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.777428 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777432 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.777437 | controller | -- \' 2026-03-09 19:10:03.777441 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777445 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.777450 | controller | - ' logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.777454 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.777458 | controller | - ' logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.777463 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.777467 | controller | found' 2026-03-09 19:10:03.777471 | controller | - ' logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.777476 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.777480 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.777484 | controller | - ' ' 2026-03-09 19:10:03.777488 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.777493 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.777497 | controller | - ' ' 2026-03-09 19:10:03.777501 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.777505 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.777510 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.777514 | controller | - ' exit 1' 2026-03-09 19:10:03.777518 | controller | - ' fi' 2026-03-09 19:10:03.777523 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.777527 | controller | - ' ' 2026-03-09 19:10:03.777531 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.777535 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.777540 | controller | | base64 -d)' 2026-03-09 19:10:03.777544 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.777548 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.777552 | controller | - ' exit 1' 2026-03-09 19:10:03.777557 | controller | - ' fi' 2026-03-09 19:10:03.777561 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.777565 | controller | - ' ' 2026-03-09 19:10:03.777570 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.777575 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.777580 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.777584 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.777588 | controller | --timeout=300s' 2026-03-09 19:10:03.777593 | controller | - ' ' 2026-03-09 19:10:03.777597 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.777601 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.777608 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.777612 | controller | - ' ' 2026-03-09 19:10:03.777617 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777621 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777625 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777629 | controller | - ' ' 2026-03-09 19:10:03.777640 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.777645 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.777649 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.777654 | controller | available"' 2026-03-09 19:10:03.777658 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.777662 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.777666 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.777671 | controller | - ' exit 1' 2026-03-09 19:10:03.777675 | controller | - ' fi' 2026-03-09 19:10:03.777679 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.777684 | controller | - ' ' 2026-03-09 19:10:03.777688 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.777692 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.777696 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777701 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.777705 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777709 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.777713 | controller | -- \' 2026-03-09 19:10:03.777719 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777723 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.777728 | controller | - ' logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.777732 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.777736 | controller | - ' logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.777740 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.777745 | controller | found' 2026-03-09 19:10:03.777749 | controller | - ' logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.777753 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.777758 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.777762 | controller | - ' ' 2026-03-09 19:10:03.777766 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.777770 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.777775 | controller | - ' ' 2026-03-09 19:10:03.777779 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.777783 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.777787 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.777793 | controller | - ' exit 1' 2026-03-09 19:10:03.777798 | controller | - ' fi' 2026-03-09 19:10:03.777802 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.777806 | controller | - ' ' 2026-03-09 19:10:03.777811 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.777815 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.777819 | controller | | base64 -d)' 2026-03-09 19:10:03.777823 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.777828 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.777832 | controller | - ' exit 1' 2026-03-09 19:10:03.777836 | controller | - ' fi' 2026-03-09 19:10:03.777840 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.777845 | controller | - ' ' 2026-03-09 19:10:03.777849 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.777853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.777857 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.777862 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.777866 | controller | --timeout=300s' 2026-03-09 19:10:03.777870 | controller | - ' ' 2026-03-09 19:10:03.777875 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.777879 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.777883 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.777887 | controller | - ' ' 2026-03-09 19:10:03.777892 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777896 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777900 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.777905 | controller | - ' ' 2026-03-09 19:10:03.777925 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.777931 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.777936 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.777940 | controller | available"' 2026-03-09 19:10:03.777944 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.777949 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.777953 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.777957 | controller | - ' exit 1' 2026-03-09 19:10:03.777961 | controller | - ' fi' 2026-03-09 19:10:03.777966 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.777972 | controller | - ' ' 2026-03-09 19:10:03.777976 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.777981 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.777985 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.777989 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.777994 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778000 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.778004 | controller | -- \' 2026-03-09 19:10:03.778008 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778013 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.778017 | controller | - ' logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.778021 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.778025 | controller | - ' logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.778030 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.778034 | controller | found' 2026-03-09 19:10:03.778038 | controller | - ' logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.778043 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.778047 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.778051 | controller | - ' ' 2026-03-09 19:10:03.778055 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.778060 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.778064 | controller | - ' ' 2026-03-09 19:10:03.778068 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.778073 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.778077 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.778081 | controller | - ' exit 1' 2026-03-09 19:10:03.778086 | controller | - ' fi' 2026-03-09 19:10:03.778090 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.778094 | controller | - ' ' 2026-03-09 19:10:03.778098 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.778103 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.778107 | controller | | base64 -d)' 2026-03-09 19:10:03.778112 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.778116 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.778120 | controller | - ' exit 1' 2026-03-09 19:10:03.778128 | controller | - ' fi' 2026-03-09 19:10:03.778133 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.778138 | controller | - ' ' 2026-03-09 19:10:03.778143 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.778161 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.778165 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.778170 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.778174 | controller | --timeout=300s' 2026-03-09 19:10:03.778178 | controller | - ' ' 2026-03-09 19:10:03.778182 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.778187 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.778191 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.778195 | controller | - ' ' 2026-03-09 19:10:03.778199 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778204 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778208 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778214 | controller | - ' ' 2026-03-09 19:10:03.778219 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.778230 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.778234 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.778239 | controller | available"' 2026-03-09 19:10:03.778243 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.778248 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.778252 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.778256 | controller | - ' exit 1' 2026-03-09 19:10:03.778260 | controller | - ' fi' 2026-03-09 19:10:03.778265 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.778269 | controller | - ' ' 2026-03-09 19:10:03.778273 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.778278 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.778282 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778286 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.778290 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778295 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.778299 | controller | -- \' 2026-03-09 19:10:03.778303 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778308 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.778312 | controller | - ' logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.778316 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.778320 | controller | - ' logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.778325 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.778329 | controller | found' 2026-03-09 19:10:03.778333 | controller | - ' logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.778337 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.778342 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.778346 | controller | - ' ' 2026-03-09 19:10:03.778350 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.778355 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.778359 | controller | - ' ' 2026-03-09 19:10:03.778363 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.778367 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.778372 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.778376 | controller | - ' exit 1' 2026-03-09 19:10:03.778380 | controller | - ' fi' 2026-03-09 19:10:03.778384 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.778389 | controller | - ' ' 2026-03-09 19:10:03.778393 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.778397 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.778401 | controller | | base64 -d)' 2026-03-09 19:10:03.778406 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.778412 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.778416 | controller | - ' exit 1' 2026-03-09 19:10:03.778420 | controller | - ' fi' 2026-03-09 19:10:03.778425 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.778429 | controller | - ' ' 2026-03-09 19:10:03.778433 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.778437 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.778442 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.778446 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.778450 | controller | --timeout=300s' 2026-03-09 19:10:03.778455 | controller | - ' ' 2026-03-09 19:10:03.778459 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.778463 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.778467 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.778472 | controller | - ' ' 2026-03-09 19:10:03.778476 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778480 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778484 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778489 | controller | - ' ' 2026-03-09 19:10:03.778493 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.778497 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.778512 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.778516 | controller | available"' 2026-03-09 19:10:03.778521 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.778525 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.778529 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.778533 | controller | - ' exit 1' 2026-03-09 19:10:03.778538 | controller | - ' fi' 2026-03-09 19:10:03.778542 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.778546 | controller | - ' ' 2026-03-09 19:10:03.778551 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.778555 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.778559 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778563 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.778568 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778572 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.778576 | controller | -- \' 2026-03-09 19:10:03.778581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778585 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.778591 | controller | - ' logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.778595 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.778601 | controller | - ' logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.778605 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.778610 | controller | found' 2026-03-09 19:10:03.778614 | controller | - ' logger.go:42: 18:57:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.778618 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.778623 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.778627 | controller | - ' ' 2026-03-09 19:10:03.778631 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.778636 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.778640 | controller | - ' ' 2026-03-09 19:10:03.778644 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.778648 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.778653 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.778657 | controller | - ' exit 1' 2026-03-09 19:10:03.778661 | controller | - ' fi' 2026-03-09 19:10:03.778666 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.778670 | controller | - ' ' 2026-03-09 19:10:03.778674 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.778678 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.778683 | controller | | base64 -d)' 2026-03-09 19:10:03.778687 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.778691 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.778696 | controller | - ' exit 1' 2026-03-09 19:10:03.778700 | controller | - ' fi' 2026-03-09 19:10:03.778704 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.778708 | controller | - ' ' 2026-03-09 19:10:03.778713 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.778717 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.778721 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.778726 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.778730 | controller | --timeout=300s' 2026-03-09 19:10:03.778736 | controller | - ' ' 2026-03-09 19:10:03.778740 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.778744 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.778748 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.778753 | controller | - ' ' 2026-03-09 19:10:03.778757 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778761 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778766 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.778770 | controller | - ' ' 2026-03-09 19:10:03.778774 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.778778 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.778790 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.778794 | controller | available"' 2026-03-09 19:10:03.778798 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.778804 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.778809 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.778813 | controller | - ' exit 1' 2026-03-09 19:10:03.778817 | controller | - ' fi' 2026-03-09 19:10:03.778821 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.778826 | controller | - ' ' 2026-03-09 19:10:03.778830 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.778834 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.778839 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778843 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.778847 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778851 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.778856 | controller | -- \' 2026-03-09 19:10:03.778860 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.778864 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.778868 | controller | - ' logger.go:42: 18:57:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.778873 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.778877 | controller | - ' logger.go:42: 18:57:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.778881 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.778886 | controller | found' 2026-03-09 19:10:03.778890 | controller | - ' logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.778894 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.778899 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.778903 | controller | - ' ' 2026-03-09 19:10:03.778907 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.778911 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.778916 | controller | - ' ' 2026-03-09 19:10:03.778920 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.778924 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.778928 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.778933 | controller | - ' exit 1' 2026-03-09 19:10:03.778937 | controller | - ' fi' 2026-03-09 19:10:03.778941 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.778946 | controller | - ' ' 2026-03-09 19:10:03.778950 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.778954 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.778958 | controller | | base64 -d)' 2026-03-09 19:10:03.778963 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.778967 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.778971 | controller | - ' exit 1' 2026-03-09 19:10:03.778976 | controller | - ' fi' 2026-03-09 19:10:03.778980 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.778984 | controller | - ' ' 2026-03-09 19:10:03.778988 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.778993 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.778998 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.779003 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.779007 | controller | --timeout=300s' 2026-03-09 19:10:03.779011 | controller | - ' ' 2026-03-09 19:10:03.779016 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.779020 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.779026 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.779030 | controller | - ' ' 2026-03-09 19:10:03.779034 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779038 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779043 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779047 | controller | - ' ' 2026-03-09 19:10:03.779051 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.779055 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.779060 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.779064 | controller | available"' 2026-03-09 19:10:03.779076 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.779081 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.779085 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.779090 | controller | - ' exit 1' 2026-03-09 19:10:03.779094 | controller | - ' fi' 2026-03-09 19:10:03.779098 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.779103 | controller | - ' ' 2026-03-09 19:10:03.779107 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.779111 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.779115 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779120 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.779124 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779128 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.779132 | controller | -- \' 2026-03-09 19:10:03.779137 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779141 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.779156 | controller | - ' logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.779162 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.779167 | controller | - ' logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.779171 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.779175 | controller | found' 2026-03-09 19:10:03.779179 | controller | - ' logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.779184 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.779188 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.779192 | controller | - ' ' 2026-03-09 19:10:03.779198 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.779203 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.779207 | controller | - ' ' 2026-03-09 19:10:03.779213 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.779217 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.779222 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.779226 | controller | - ' exit 1' 2026-03-09 19:10:03.779230 | controller | - ' fi' 2026-03-09 19:10:03.779234 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.779239 | controller | - ' ' 2026-03-09 19:10:03.779245 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.779249 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.779253 | controller | | base64 -d)' 2026-03-09 19:10:03.779258 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.779262 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.779266 | controller | - ' exit 1' 2026-03-09 19:10:03.779270 | controller | - ' fi' 2026-03-09 19:10:03.779275 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.779279 | controller | - ' ' 2026-03-09 19:10:03.779283 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.779287 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.779292 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.779296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.779300 | controller | --timeout=300s' 2026-03-09 19:10:03.779305 | controller | - ' ' 2026-03-09 19:10:03.779309 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.779313 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.779317 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.779322 | controller | - ' ' 2026-03-09 19:10:03.779326 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779330 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779334 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779339 | controller | - ' ' 2026-03-09 19:10:03.779343 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.779347 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.779351 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.779356 | controller | available"' 2026-03-09 19:10:03.779367 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.779372 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.779376 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.779381 | controller | - ' exit 1' 2026-03-09 19:10:03.779385 | controller | - ' fi' 2026-03-09 19:10:03.779389 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.779394 | controller | - ' ' 2026-03-09 19:10:03.779398 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.779404 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.779408 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779412 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.779417 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779421 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.779425 | controller | -- \' 2026-03-09 19:10:03.779429 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779434 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.779438 | controller | - ' logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.779442 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.779446 | controller | - ' logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.779451 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.779455 | controller | found' 2026-03-09 19:10:03.779459 | controller | - ' logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.779463 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.779468 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.779472 | controller | - ' ' 2026-03-09 19:10:03.779476 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.779480 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.779485 | controller | - ' ' 2026-03-09 19:10:03.779489 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.779493 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.779498 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.779502 | controller | - ' exit 1' 2026-03-09 19:10:03.779506 | controller | - ' fi' 2026-03-09 19:10:03.779510 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.779515 | controller | - ' ' 2026-03-09 19:10:03.779519 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.779523 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.779527 | controller | | base64 -d)' 2026-03-09 19:10:03.779532 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.779536 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.779540 | controller | - ' exit 1' 2026-03-09 19:10:03.779544 | controller | - ' fi' 2026-03-09 19:10:03.779549 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.779553 | controller | - ' ' 2026-03-09 19:10:03.779557 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.779561 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.779566 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.779570 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.779574 | controller | --timeout=300s' 2026-03-09 19:10:03.779578 | controller | - ' ' 2026-03-09 19:10:03.779583 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.779587 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.779593 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.779597 | controller | - ' ' 2026-03-09 19:10:03.779601 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779606 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779610 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779614 | controller | - ' ' 2026-03-09 19:10:03.779618 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.779623 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.779627 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.779631 | controller | available"' 2026-03-09 19:10:03.779636 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.779646 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.779670 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.779676 | controller | - ' exit 1' 2026-03-09 19:10:03.779681 | controller | - ' fi' 2026-03-09 19:10:03.779685 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.779690 | controller | - ' ' 2026-03-09 19:10:03.779696 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.779701 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.779705 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779709 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.779713 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779718 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.779722 | controller | -- \' 2026-03-09 19:10:03.779726 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779730 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.779735 | controller | - ' logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.779739 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.779743 | controller | - ' logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.779747 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.779752 | controller | found' 2026-03-09 19:10:03.779756 | controller | - ' logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.779760 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.779764 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.779769 | controller | - ' ' 2026-03-09 19:10:03.779773 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.779777 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.779781 | controller | - ' ' 2026-03-09 19:10:03.779786 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.779790 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.779794 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.779799 | controller | - ' exit 1' 2026-03-09 19:10:03.779805 | controller | - ' fi' 2026-03-09 19:10:03.779809 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.779813 | controller | - ' ' 2026-03-09 19:10:03.779818 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.779822 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.779826 | controller | | base64 -d)' 2026-03-09 19:10:03.779831 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.779835 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.779839 | controller | - ' exit 1' 2026-03-09 19:10:03.779843 | controller | - ' fi' 2026-03-09 19:10:03.779848 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.779852 | controller | - ' ' 2026-03-09 19:10:03.779856 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.779860 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.779865 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.779869 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.779873 | controller | --timeout=300s' 2026-03-09 19:10:03.779877 | controller | - ' ' 2026-03-09 19:10:03.779882 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.779886 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.779890 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.779895 | controller | - ' ' 2026-03-09 19:10:03.779902 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779906 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779910 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.779915 | controller | - ' ' 2026-03-09 19:10:03.779919 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.779923 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.779928 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.779934 | controller | available"' 2026-03-09 19:10:03.779938 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.779942 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.779956 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.779961 | controller | - ' exit 1' 2026-03-09 19:10:03.779965 | controller | - ' fi' 2026-03-09 19:10:03.779970 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.779974 | controller | - ' ' 2026-03-09 19:10:03.779978 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.779982 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.779987 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.779991 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.779997 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780001 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.780008 | controller | -- \' 2026-03-09 19:10:03.780012 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780016 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.780021 | controller | - ' logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.780025 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.780029 | controller | - ' logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.780034 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.780038 | controller | found' 2026-03-09 19:10:03.780042 | controller | - ' logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.780046 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.780051 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.780055 | controller | - ' ' 2026-03-09 19:10:03.780059 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.780063 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.780068 | controller | - ' ' 2026-03-09 19:10:03.780072 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.780076 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.780080 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.780085 | controller | - ' exit 1' 2026-03-09 19:10:03.780089 | controller | - ' fi' 2026-03-09 19:10:03.780093 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.780098 | controller | - ' ' 2026-03-09 19:10:03.780102 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.780106 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.780110 | controller | | base64 -d)' 2026-03-09 19:10:03.780115 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.780119 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.780123 | controller | - ' exit 1' 2026-03-09 19:10:03.780127 | controller | - ' fi' 2026-03-09 19:10:03.780132 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.780136 | controller | - ' ' 2026-03-09 19:10:03.780140 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.780156 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.780162 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.780167 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.780171 | controller | --timeout=300s' 2026-03-09 19:10:03.780175 | controller | - ' ' 2026-03-09 19:10:03.780179 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.780184 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.780188 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.780192 | controller | - ' ' 2026-03-09 19:10:03.780196 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780201 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780205 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780211 | controller | - ' ' 2026-03-09 19:10:03.780215 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.780220 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.780224 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.780228 | controller | available"' 2026-03-09 19:10:03.780234 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.780239 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.780243 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.780255 | controller | - ' exit 1' 2026-03-09 19:10:03.780259 | controller | - ' fi' 2026-03-09 19:10:03.780264 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.780268 | controller | - ' ' 2026-03-09 19:10:03.780272 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.780277 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.780281 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780287 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.780291 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780295 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.780300 | controller | -- \' 2026-03-09 19:10:03.780304 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780308 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.780312 | controller | - ' logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.780317 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.780321 | controller | - ' logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.780325 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.780329 | controller | found' 2026-03-09 19:10:03.780334 | controller | - ' logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.780338 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.780342 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.780347 | controller | - ' ' 2026-03-09 19:10:03.780351 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.780355 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.780359 | controller | - ' ' 2026-03-09 19:10:03.780364 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.780368 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.780372 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.780376 | controller | - ' exit 1' 2026-03-09 19:10:03.780381 | controller | - ' fi' 2026-03-09 19:10:03.780385 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.780389 | controller | - ' ' 2026-03-09 19:10:03.780394 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.780398 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.780402 | controller | | base64 -d)' 2026-03-09 19:10:03.780406 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.780411 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.780416 | controller | - ' exit 1' 2026-03-09 19:10:03.780421 | controller | - ' fi' 2026-03-09 19:10:03.780425 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.780429 | controller | - ' ' 2026-03-09 19:10:03.780434 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.780438 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.780442 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.780446 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.780451 | controller | --timeout=300s' 2026-03-09 19:10:03.780455 | controller | - ' ' 2026-03-09 19:10:03.780459 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.780464 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.780468 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.780472 | controller | - ' ' 2026-03-09 19:10:03.780476 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780481 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780485 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780489 | controller | - ' ' 2026-03-09 19:10:03.780494 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.780498 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.780502 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.780506 | controller | available"' 2026-03-09 19:10:03.780511 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.780516 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.780521 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.780525 | controller | - ' exit 1' 2026-03-09 19:10:03.780529 | controller | - ' fi' 2026-03-09 19:10:03.780540 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.780544 | controller | - ' ' 2026-03-09 19:10:03.780549 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.780553 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.780557 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780561 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.780566 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780570 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.780574 | controller | -- \' 2026-03-09 19:10:03.780579 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780583 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.780587 | controller | - ' logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.780591 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.780596 | controller | - ' logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.780602 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.780606 | controller | found' 2026-03-09 19:10:03.780610 | controller | - ' logger.go:42: 18:57:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.780615 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.780619 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.780623 | controller | - ' ' 2026-03-09 19:10:03.780627 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.780632 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.780636 | controller | - ' ' 2026-03-09 19:10:03.780640 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.780645 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.780649 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.780653 | controller | - ' exit 1' 2026-03-09 19:10:03.780657 | controller | - ' fi' 2026-03-09 19:10:03.780662 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.780666 | controller | - ' ' 2026-03-09 19:10:03.780670 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.780675 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.780679 | controller | | base64 -d)' 2026-03-09 19:10:03.780683 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.780687 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.780692 | controller | - ' exit 1' 2026-03-09 19:10:03.780696 | controller | - ' fi' 2026-03-09 19:10:03.780700 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.780704 | controller | - ' ' 2026-03-09 19:10:03.780709 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.780713 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.780717 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.780722 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.780726 | controller | --timeout=300s' 2026-03-09 19:10:03.780730 | controller | - ' ' 2026-03-09 19:10:03.780734 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.780739 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.780743 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.780747 | controller | - ' ' 2026-03-09 19:10:03.780752 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780756 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780760 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.780765 | controller | - ' ' 2026-03-09 19:10:03.780769 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.780773 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.780777 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.780782 | controller | available"' 2026-03-09 19:10:03.780786 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.780790 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.780796 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.780800 | controller | - ' exit 1' 2026-03-09 19:10:03.780805 | controller | - ' fi' 2026-03-09 19:10:03.780816 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.780820 | controller | - ' ' 2026-03-09 19:10:03.780825 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.780829 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.780833 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780838 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.780842 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780846 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.780850 | controller | -- \' 2026-03-09 19:10:03.780855 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.780861 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.780865 | controller | - ' logger.go:42: 18:57:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.780869 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.780875 | controller | - ' logger.go:42: 18:57:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.780880 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.780884 | controller | found' 2026-03-09 19:10:03.780888 | controller | - ' logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.780893 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.780897 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.780902 | controller | - ' ' 2026-03-09 19:10:03.780906 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.780910 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.780914 | controller | - ' ' 2026-03-09 19:10:03.780919 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.780923 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.780927 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.780931 | controller | - ' exit 1' 2026-03-09 19:10:03.780936 | controller | - ' fi' 2026-03-09 19:10:03.780940 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.780944 | controller | - ' ' 2026-03-09 19:10:03.780949 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.780953 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.780957 | controller | | base64 -d)' 2026-03-09 19:10:03.780961 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.780966 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.780970 | controller | - ' exit 1' 2026-03-09 19:10:03.780974 | controller | - ' fi' 2026-03-09 19:10:03.780978 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.780983 | controller | - ' ' 2026-03-09 19:10:03.780987 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.780991 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.780996 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.781001 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.781006 | controller | --timeout=300s' 2026-03-09 19:10:03.781010 | controller | - ' ' 2026-03-09 19:10:03.781014 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.781019 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.781023 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.781027 | controller | - ' ' 2026-03-09 19:10:03.781031 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781037 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781042 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781046 | controller | - ' ' 2026-03-09 19:10:03.781050 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.781054 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.781059 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.781063 | controller | available"' 2026-03-09 19:10:03.781067 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.781071 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.781076 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.781080 | controller | - ' exit 1' 2026-03-09 19:10:03.781084 | controller | - ' fi' 2026-03-09 19:10:03.781088 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.781093 | controller | - ' ' 2026-03-09 19:10:03.781104 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.781109 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.781114 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781120 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.781124 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781129 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.781134 | controller | -- \' 2026-03-09 19:10:03.781139 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781143 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.781160 | controller | - ' logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.781165 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.781169 | controller | - ' logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.781173 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.781177 | controller | found' 2026-03-09 19:10:03.781182 | controller | - ' logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.781186 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.781190 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.781194 | controller | - ' ' 2026-03-09 19:10:03.781199 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.781206 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.781210 | controller | - ' ' 2026-03-09 19:10:03.781215 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.781219 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.781223 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.781228 | controller | - ' exit 1' 2026-03-09 19:10:03.781232 | controller | - ' fi' 2026-03-09 19:10:03.781236 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.781240 | controller | - ' ' 2026-03-09 19:10:03.781245 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.781249 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.781253 | controller | | base64 -d)' 2026-03-09 19:10:03.781258 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.781264 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.781268 | controller | - ' exit 1' 2026-03-09 19:10:03.781272 | controller | - ' fi' 2026-03-09 19:10:03.781277 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.781281 | controller | - ' ' 2026-03-09 19:10:03.781287 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.781291 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.781295 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.781300 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.781304 | controller | --timeout=300s' 2026-03-09 19:10:03.781308 | controller | - ' ' 2026-03-09 19:10:03.781313 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.781317 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.781321 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.781325 | controller | - ' ' 2026-03-09 19:10:03.781330 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781334 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781338 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781343 | controller | - ' ' 2026-03-09 19:10:03.781347 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.781351 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.781355 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.781360 | controller | available"' 2026-03-09 19:10:03.781364 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.781368 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.781372 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.781377 | controller | - ' exit 1' 2026-03-09 19:10:03.781381 | controller | - ' fi' 2026-03-09 19:10:03.781385 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.781389 | controller | - ' ' 2026-03-09 19:10:03.781402 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.781406 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.781412 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781417 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.781421 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781425 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.781429 | controller | -- \' 2026-03-09 19:10:03.781434 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781438 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.781442 | controller | - ' logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.781446 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.781452 | controller | - ' logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.781457 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.781461 | controller | found' 2026-03-09 19:10:03.781465 | controller | - ' logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.781469 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.781474 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.781478 | controller | - ' ' 2026-03-09 19:10:03.781482 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.781487 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.781491 | controller | - ' ' 2026-03-09 19:10:03.781495 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.781499 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.781504 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.781508 | controller | - ' exit 1' 2026-03-09 19:10:03.781512 | controller | - ' fi' 2026-03-09 19:10:03.781516 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.781521 | controller | - ' ' 2026-03-09 19:10:03.781525 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.781529 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.781533 | controller | | base64 -d)' 2026-03-09 19:10:03.781538 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.781542 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.781546 | controller | - ' exit 1' 2026-03-09 19:10:03.781550 | controller | - ' fi' 2026-03-09 19:10:03.781555 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.781559 | controller | - ' ' 2026-03-09 19:10:03.781563 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.781567 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.781572 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.781576 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.781580 | controller | --timeout=300s' 2026-03-09 19:10:03.781585 | controller | - ' ' 2026-03-09 19:10:03.781589 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.781593 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.781597 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.781604 | controller | - ' ' 2026-03-09 19:10:03.781609 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781613 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781617 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781621 | controller | - ' ' 2026-03-09 19:10:03.781626 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.781630 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.781634 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.781638 | controller | available"' 2026-03-09 19:10:03.781643 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.781647 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.781651 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.781656 | controller | - ' exit 1' 2026-03-09 19:10:03.781660 | controller | - ' fi' 2026-03-09 19:10:03.781664 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.781668 | controller | - ' ' 2026-03-09 19:10:03.781673 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.781684 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.781688 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781692 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.781697 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781701 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.781705 | controller | -- \' 2026-03-09 19:10:03.781710 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781714 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.781718 | controller | - ' logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.781722 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.781727 | controller | - ' logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.781731 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.781735 | controller | found' 2026-03-09 19:10:03.781740 | controller | - ' logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.781744 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.781748 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.781752 | controller | - ' ' 2026-03-09 19:10:03.781757 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.781761 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.781765 | controller | - ' ' 2026-03-09 19:10:03.781769 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.781774 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.781778 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.781782 | controller | - ' exit 1' 2026-03-09 19:10:03.781786 | controller | - ' fi' 2026-03-09 19:10:03.781792 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.781797 | controller | - ' ' 2026-03-09 19:10:03.781803 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.781807 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.781811 | controller | | base64 -d)' 2026-03-09 19:10:03.781816 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.781820 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.781824 | controller | - ' exit 1' 2026-03-09 19:10:03.781828 | controller | - ' fi' 2026-03-09 19:10:03.781833 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.781837 | controller | - ' ' 2026-03-09 19:10:03.781841 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.781845 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.781850 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.781854 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.781858 | controller | --timeout=300s' 2026-03-09 19:10:03.781862 | controller | - ' ' 2026-03-09 19:10:03.781867 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.781871 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.781875 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.781880 | controller | - ' ' 2026-03-09 19:10:03.781884 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781888 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781892 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.781897 | controller | - ' ' 2026-03-09 19:10:03.781901 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.781905 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.781909 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.781914 | controller | available"' 2026-03-09 19:10:03.781918 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.781922 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.781926 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.781931 | controller | - ' exit 1' 2026-03-09 19:10:03.781935 | controller | - ' fi' 2026-03-09 19:10:03.781939 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.781943 | controller | - ' ' 2026-03-09 19:10:03.781948 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.781952 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.781962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781967 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.781971 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781976 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.781983 | controller | -- \' 2026-03-09 19:10:03.781988 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.781992 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.781996 | controller | - ' logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.782001 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.782005 | controller | - ' logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.782011 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.782015 | controller | found' 2026-03-09 19:10:03.782019 | controller | - ' logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.782023 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.782028 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.782032 | controller | - ' ' 2026-03-09 19:10:03.782036 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.782040 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.782045 | controller | - ' ' 2026-03-09 19:10:03.782049 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.782053 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.782058 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.782062 | controller | - ' exit 1' 2026-03-09 19:10:03.782066 | controller | - ' fi' 2026-03-09 19:10:03.782070 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.782075 | controller | - ' ' 2026-03-09 19:10:03.782079 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.782083 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.782087 | controller | | base64 -d)' 2026-03-09 19:10:03.782092 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.782096 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.782100 | controller | - ' exit 1' 2026-03-09 19:10:03.782105 | controller | - ' fi' 2026-03-09 19:10:03.782109 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.782113 | controller | - ' ' 2026-03-09 19:10:03.782117 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.782122 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.782126 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.782130 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.782134 | controller | --timeout=300s' 2026-03-09 19:10:03.782139 | controller | - ' ' 2026-03-09 19:10:03.782143 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.782160 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.782164 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.782168 | controller | - ' ' 2026-03-09 19:10:03.782172 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782177 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782181 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782185 | controller | - ' ' 2026-03-09 19:10:03.782191 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.782196 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.782200 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.782204 | controller | available"' 2026-03-09 19:10:03.782208 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.782213 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.782217 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.782221 | controller | - ' exit 1' 2026-03-09 19:10:03.782226 | controller | - ' fi' 2026-03-09 19:10:03.782230 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.782234 | controller | - ' ' 2026-03-09 19:10:03.782238 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.782243 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.782254 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782259 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.782263 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782267 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.782272 | controller | -- \' 2026-03-09 19:10:03.782276 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782282 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.782286 | controller | - ' logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.782291 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.782295 | controller | - ' logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.782299 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.782303 | controller | found' 2026-03-09 19:10:03.782308 | controller | - ' logger.go:42: 18:57:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.782312 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.782316 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.782320 | controller | - ' ' 2026-03-09 19:10:03.782325 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.782329 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.782333 | controller | - ' ' 2026-03-09 19:10:03.782337 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.782341 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.782346 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.782350 | controller | - ' exit 1' 2026-03-09 19:10:03.782354 | controller | - ' fi' 2026-03-09 19:10:03.782360 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.782364 | controller | - ' ' 2026-03-09 19:10:03.782369 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.782373 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.782377 | controller | | base64 -d)' 2026-03-09 19:10:03.782381 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.782386 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.782391 | controller | - ' exit 1' 2026-03-09 19:10:03.782396 | controller | - ' fi' 2026-03-09 19:10:03.782400 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.782404 | controller | - ' ' 2026-03-09 19:10:03.782409 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.782413 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.782417 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.782421 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.782426 | controller | --timeout=300s' 2026-03-09 19:10:03.782430 | controller | - ' ' 2026-03-09 19:10:03.782434 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.782438 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.782443 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.782447 | controller | - ' ' 2026-03-09 19:10:03.782451 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782455 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782460 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782464 | controller | - ' ' 2026-03-09 19:10:03.782468 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.782472 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.782477 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.782481 | controller | available"' 2026-03-09 19:10:03.782485 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.782489 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.782494 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.782498 | controller | - ' exit 1' 2026-03-09 19:10:03.782502 | controller | - ' fi' 2026-03-09 19:10:03.782506 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.782511 | controller | - ' ' 2026-03-09 19:10:03.782515 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.782519 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.782530 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782535 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.782539 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782543 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.782547 | controller | -- \' 2026-03-09 19:10:03.782552 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782556 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.782563 | controller | - ' logger.go:42: 18:57:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.782567 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.782572 | controller | - ' logger.go:42: 18:58:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.782578 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.782582 | controller | found' 2026-03-09 19:10:03.782588 | controller | - ' logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.782592 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.782597 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.782601 | controller | - ' ' 2026-03-09 19:10:03.782605 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.782609 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.782614 | controller | - ' ' 2026-03-09 19:10:03.782618 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.782622 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.782626 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.782631 | controller | - ' exit 1' 2026-03-09 19:10:03.782635 | controller | - ' fi' 2026-03-09 19:10:03.782639 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.782643 | controller | - ' ' 2026-03-09 19:10:03.782648 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.782652 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.782656 | controller | | base64 -d)' 2026-03-09 19:10:03.782660 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.782665 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.782669 | controller | - ' exit 1' 2026-03-09 19:10:03.782673 | controller | - ' fi' 2026-03-09 19:10:03.782677 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.782682 | controller | - ' ' 2026-03-09 19:10:03.782686 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.782690 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.782694 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.782699 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.782703 | controller | --timeout=300s' 2026-03-09 19:10:03.782709 | controller | - ' ' 2026-03-09 19:10:03.782713 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.782717 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.782721 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.782726 | controller | - ' ' 2026-03-09 19:10:03.782730 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782734 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782739 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.782743 | controller | - ' ' 2026-03-09 19:10:03.782747 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.782751 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.782755 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.782760 | controller | available"' 2026-03-09 19:10:03.782764 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.782768 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.782774 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.782778 | controller | - ' exit 1' 2026-03-09 19:10:03.782783 | controller | - ' fi' 2026-03-09 19:10:03.782787 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.782791 | controller | - ' ' 2026-03-09 19:10:03.782796 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.782800 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.782804 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782815 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.782820 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782824 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.782828 | controller | -- \' 2026-03-09 19:10:03.782833 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.782837 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.782841 | controller | - ' logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.782846 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.782850 | controller | - ' logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.782854 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.782859 | controller | found' 2026-03-09 19:10:03.782863 | controller | - ' logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.782867 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.782872 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.782876 | controller | - ' ' 2026-03-09 19:10:03.782880 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.782884 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.782888 | controller | - ' ' 2026-03-09 19:10:03.782893 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.782897 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.782901 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.782905 | controller | - ' exit 1' 2026-03-09 19:10:03.782910 | controller | - ' fi' 2026-03-09 19:10:03.782914 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.782918 | controller | - ' ' 2026-03-09 19:10:03.782923 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.782927 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.782931 | controller | | base64 -d)' 2026-03-09 19:10:03.782935 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.782940 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.782944 | controller | - ' exit 1' 2026-03-09 19:10:03.782948 | controller | - ' fi' 2026-03-09 19:10:03.782952 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.782957 | controller | - ' ' 2026-03-09 19:10:03.782961 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.782965 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.782970 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.782974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.782980 | controller | --timeout=300s' 2026-03-09 19:10:03.782984 | controller | - ' ' 2026-03-09 19:10:03.782989 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.782993 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.782997 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.783001 | controller | - ' ' 2026-03-09 19:10:03.783006 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783010 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783016 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783020 | controller | - ' ' 2026-03-09 19:10:03.783024 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.783029 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.783033 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.783037 | controller | available"' 2026-03-09 19:10:03.783041 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.783047 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.783052 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.783056 | controller | - ' exit 1' 2026-03-09 19:10:03.783060 | controller | - ' fi' 2026-03-09 19:10:03.783064 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.783069 | controller | - ' ' 2026-03-09 19:10:03.783073 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.783077 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.783081 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783086 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.783097 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783101 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.783106 | controller | -- \' 2026-03-09 19:10:03.783110 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783114 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.783119 | controller | - ' logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.783123 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.783127 | controller | - ' logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.783131 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.783136 | controller | found' 2026-03-09 19:10:03.783140 | controller | - ' logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.783154 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.783160 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.783165 | controller | - ' ' 2026-03-09 19:10:03.783175 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.783180 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.783186 | controller | - ' ' 2026-03-09 19:10:03.783190 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.783194 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.783199 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.783203 | controller | - ' exit 1' 2026-03-09 19:10:03.783207 | controller | - ' fi' 2026-03-09 19:10:03.783211 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.783216 | controller | - ' ' 2026-03-09 19:10:03.783220 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.783224 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.783228 | controller | | base64 -d)' 2026-03-09 19:10:03.783233 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.783237 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.783241 | controller | - ' exit 1' 2026-03-09 19:10:03.783245 | controller | - ' fi' 2026-03-09 19:10:03.783250 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.783254 | controller | - ' ' 2026-03-09 19:10:03.783258 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.783263 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.783267 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.783271 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.783275 | controller | --timeout=300s' 2026-03-09 19:10:03.783280 | controller | - ' ' 2026-03-09 19:10:03.783284 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.783288 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.783292 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.783297 | controller | - ' ' 2026-03-09 19:10:03.783301 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783305 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783309 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783314 | controller | - ' ' 2026-03-09 19:10:03.783318 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.783322 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.783326 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.783331 | controller | available"' 2026-03-09 19:10:03.783335 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.783339 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.783343 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.783348 | controller | - ' exit 1' 2026-03-09 19:10:03.783352 | controller | - ' fi' 2026-03-09 19:10:03.783358 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.783362 | controller | - ' ' 2026-03-09 19:10:03.783366 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.783371 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.783375 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783388 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.783400 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783404 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.783409 | controller | -- \' 2026-03-09 19:10:03.783413 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783418 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.783422 | controller | - ' logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.783426 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.783431 | controller | - ' logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.783435 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.783439 | controller | found' 2026-03-09 19:10:03.783443 | controller | - ' logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.783447 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.783452 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.783456 | controller | - ' ' 2026-03-09 19:10:03.783460 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.783465 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.783469 | controller | - ' ' 2026-03-09 19:10:03.783473 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.783477 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.783482 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.783486 | controller | - ' exit 1' 2026-03-09 19:10:03.783490 | controller | - ' fi' 2026-03-09 19:10:03.783494 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.783499 | controller | - ' ' 2026-03-09 19:10:03.783503 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.783507 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.783512 | controller | | base64 -d)' 2026-03-09 19:10:03.783516 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.783520 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.783524 | controller | - ' exit 1' 2026-03-09 19:10:03.783529 | controller | - ' fi' 2026-03-09 19:10:03.783533 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.783537 | controller | - ' ' 2026-03-09 19:10:03.783541 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.783546 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.783550 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.783554 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.783558 | controller | --timeout=300s' 2026-03-09 19:10:03.783563 | controller | - ' ' 2026-03-09 19:10:03.783567 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.783571 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.783575 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.783582 | controller | - ' ' 2026-03-09 19:10:03.783586 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783590 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783594 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783599 | controller | - ' ' 2026-03-09 19:10:03.783603 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.783607 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.783611 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.783616 | controller | available"' 2026-03-09 19:10:03.783620 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.783624 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.783628 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.783633 | controller | - ' exit 1' 2026-03-09 19:10:03.783637 | controller | - ' fi' 2026-03-09 19:10:03.783641 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.783668 | controller | - ' ' 2026-03-09 19:10:03.783673 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.783677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.783681 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783686 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.783697 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783701 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.783705 | controller | -- \' 2026-03-09 19:10:03.783710 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783714 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.783718 | controller | - ' logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.783723 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.783727 | controller | - ' logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.783731 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.783735 | controller | found' 2026-03-09 19:10:03.783740 | controller | - ' logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.783744 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.783748 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.783752 | controller | - ' ' 2026-03-09 19:10:03.783757 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.783763 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.783767 | controller | - ' ' 2026-03-09 19:10:03.783772 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.783776 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.783780 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.783784 | controller | - ' exit 1' 2026-03-09 19:10:03.783789 | controller | - ' fi' 2026-03-09 19:10:03.783793 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.783799 | controller | - ' ' 2026-03-09 19:10:03.783803 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.783808 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.783812 | controller | | base64 -d)' 2026-03-09 19:10:03.783816 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.783821 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.783825 | controller | - ' exit 1' 2026-03-09 19:10:03.783829 | controller | - ' fi' 2026-03-09 19:10:03.783833 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.783838 | controller | - ' ' 2026-03-09 19:10:03.783842 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.783846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.783851 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.783855 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.783859 | controller | --timeout=300s' 2026-03-09 19:10:03.783863 | controller | - ' ' 2026-03-09 19:10:03.783868 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.783872 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.783876 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.783880 | controller | - ' ' 2026-03-09 19:10:03.783888 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783892 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783896 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.783900 | controller | - ' ' 2026-03-09 19:10:03.783905 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.783909 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.783913 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.783917 | controller | available"' 2026-03-09 19:10:03.783922 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.783926 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.783930 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.783934 | controller | - ' exit 1' 2026-03-09 19:10:03.783939 | controller | - ' fi' 2026-03-09 19:10:03.783943 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.783947 | controller | - ' ' 2026-03-09 19:10:03.783951 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.783956 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.783960 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783964 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.783968 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783979 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.783984 | controller | -- \' 2026-03-09 19:10:03.783988 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.783994 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.784000 | controller | - ' logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.784004 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.784008 | controller | - ' logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.784013 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.784017 | controller | found' 2026-03-09 19:10:03.784021 | controller | - ' logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.784026 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.784030 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.784034 | controller | - ' ' 2026-03-09 19:10:03.784038 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.784043 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.784047 | controller | - ' ' 2026-03-09 19:10:03.784051 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.784055 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.784060 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.784064 | controller | - ' exit 1' 2026-03-09 19:10:03.784068 | controller | - ' fi' 2026-03-09 19:10:03.784072 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.784077 | controller | - ' ' 2026-03-09 19:10:03.784081 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.784085 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.784089 | controller | | base64 -d)' 2026-03-09 19:10:03.784094 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.784098 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.784102 | controller | - ' exit 1' 2026-03-09 19:10:03.784106 | controller | - ' fi' 2026-03-09 19:10:03.784111 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.784115 | controller | - ' ' 2026-03-09 19:10:03.784119 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.784123 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.784128 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.784132 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.784136 | controller | --timeout=300s' 2026-03-09 19:10:03.784140 | controller | - ' ' 2026-03-09 19:10:03.784156 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.784163 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.784167 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.784171 | controller | - ' ' 2026-03-09 19:10:03.784175 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784180 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784184 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784188 | controller | - ' ' 2026-03-09 19:10:03.784192 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.784199 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.784203 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.784208 | controller | available"' 2026-03-09 19:10:03.784212 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.784216 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.784221 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.784225 | controller | - ' exit 1' 2026-03-09 19:10:03.784229 | controller | - ' fi' 2026-03-09 19:10:03.784233 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.784238 | controller | - ' ' 2026-03-09 19:10:03.784242 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.784246 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.784250 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784255 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.784259 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784263 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.784276 | controller | -- \' 2026-03-09 19:10:03.784280 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784285 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.784289 | controller | - ' logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.784293 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.784298 | controller | - ' logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.784302 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.784306 | controller | found' 2026-03-09 19:10:03.784311 | controller | - ' logger.go:42: 18:58:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.784315 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.784319 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.784326 | controller | - ' ' 2026-03-09 19:10:03.784330 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.784334 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.784339 | controller | - ' ' 2026-03-09 19:10:03.784343 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.784349 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.784353 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.784358 | controller | - ' exit 1' 2026-03-09 19:10:03.784362 | controller | - ' fi' 2026-03-09 19:10:03.784367 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.784371 | controller | - ' ' 2026-03-09 19:10:03.784377 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.784381 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.784386 | controller | | base64 -d)' 2026-03-09 19:10:03.784390 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.784394 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.784399 | controller | - ' exit 1' 2026-03-09 19:10:03.784405 | controller | - ' fi' 2026-03-09 19:10:03.784409 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.784413 | controller | - ' ' 2026-03-09 19:10:03.784418 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.784422 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.784426 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.784431 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.784435 | controller | --timeout=300s' 2026-03-09 19:10:03.784439 | controller | - ' ' 2026-03-09 19:10:03.784445 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.784450 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.784454 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.784458 | controller | - ' ' 2026-03-09 19:10:03.784463 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784467 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784471 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784476 | controller | - ' ' 2026-03-09 19:10:03.784480 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.784484 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.784489 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.784493 | controller | available"' 2026-03-09 19:10:03.784497 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.784502 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.784506 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.784510 | controller | - ' exit 1' 2026-03-09 19:10:03.784514 | controller | - ' fi' 2026-03-09 19:10:03.784519 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.784523 | controller | - ' ' 2026-03-09 19:10:03.784527 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.784532 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.784536 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784540 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.784544 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784549 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.784553 | controller | -- \' 2026-03-09 19:10:03.784564 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784569 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.784573 | controller | - ' logger.go:42: 18:58:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.784578 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.784582 | controller | - ' logger.go:42: 18:58:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.784586 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.784592 | controller | found' 2026-03-09 19:10:03.784597 | controller | - ' logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.784601 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.784605 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.784609 | controller | - ' ' 2026-03-09 19:10:03.784614 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.784618 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.784622 | controller | - ' ' 2026-03-09 19:10:03.784626 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.784631 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.784635 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.784641 | controller | - ' exit 1' 2026-03-09 19:10:03.784647 | controller | - ' fi' 2026-03-09 19:10:03.784653 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.784659 | controller | - ' ' 2026-03-09 19:10:03.784665 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.784671 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.784677 | controller | | base64 -d)' 2026-03-09 19:10:03.784682 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.784687 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.784691 | controller | - ' exit 1' 2026-03-09 19:10:03.784695 | controller | - ' fi' 2026-03-09 19:10:03.784700 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.784704 | controller | - ' ' 2026-03-09 19:10:03.784708 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.784712 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.784717 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.784721 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.784725 | controller | --timeout=300s' 2026-03-09 19:10:03.784729 | controller | - ' ' 2026-03-09 19:10:03.784734 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.784738 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.784742 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.784747 | controller | - ' ' 2026-03-09 19:10:03.784751 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784755 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784759 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.784764 | controller | - ' ' 2026-03-09 19:10:03.784768 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.784772 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.784776 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.784781 | controller | available"' 2026-03-09 19:10:03.784785 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.784789 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.784793 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.784800 | controller | - ' exit 1' 2026-03-09 19:10:03.784804 | controller | - ' fi' 2026-03-09 19:10:03.784808 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.784813 | controller | - ' ' 2026-03-09 19:10:03.784817 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.784821 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.784825 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784830 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.784834 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784838 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.784842 | controller | -- \' 2026-03-09 19:10:03.784854 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.784859 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.784863 | controller | - ' logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.784867 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.784872 | controller | - ' logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.784876 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.784880 | controller | found' 2026-03-09 19:10:03.784885 | controller | - ' logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.784889 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.784893 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.784897 | controller | - ' ' 2026-03-09 19:10:03.784902 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.784906 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.784910 | controller | - ' ' 2026-03-09 19:10:03.784915 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.784919 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.784923 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.784927 | controller | - ' exit 1' 2026-03-09 19:10:03.784932 | controller | - ' fi' 2026-03-09 19:10:03.784936 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.784942 | controller | - ' ' 2026-03-09 19:10:03.784947 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.784951 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.784955 | controller | | base64 -d)' 2026-03-09 19:10:03.784960 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.784964 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.784968 | controller | - ' exit 1' 2026-03-09 19:10:03.784972 | controller | - ' fi' 2026-03-09 19:10:03.784977 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.784981 | controller | - ' ' 2026-03-09 19:10:03.784985 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.784990 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.784994 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.784998 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.785005 | controller | --timeout=300s' 2026-03-09 19:10:03.785009 | controller | - ' ' 2026-03-09 19:10:03.785014 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.785018 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.785022 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.785026 | controller | - ' ' 2026-03-09 19:10:03.785031 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785035 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785039 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785044 | controller | - ' ' 2026-03-09 19:10:03.785048 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.785054 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.785058 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.785062 | controller | available"' 2026-03-09 19:10:03.785067 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.785071 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.785075 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.785080 | controller | - ' exit 1' 2026-03-09 19:10:03.785084 | controller | - ' fi' 2026-03-09 19:10:03.785088 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.785092 | controller | - ' ' 2026-03-09 19:10:03.785097 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.785101 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.785105 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785109 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.785114 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785118 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.785122 | controller | -- \' 2026-03-09 19:10:03.785127 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785138 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.785142 | controller | - ' logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.785160 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.785165 | controller | - ' logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.785170 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.785174 | controller | found' 2026-03-09 19:10:03.785178 | controller | - ' logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.785183 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.785187 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.785191 | controller | - ' ' 2026-03-09 19:10:03.785195 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.785200 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.785207 | controller | - ' ' 2026-03-09 19:10:03.785211 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.785216 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.785220 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.785224 | controller | - ' exit 1' 2026-03-09 19:10:03.785228 | controller | - ' fi' 2026-03-09 19:10:03.785233 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.785237 | controller | - ' ' 2026-03-09 19:10:03.785241 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.785246 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.785250 | controller | | base64 -d)' 2026-03-09 19:10:03.785254 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.785260 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.785265 | controller | - ' exit 1' 2026-03-09 19:10:03.785269 | controller | - ' fi' 2026-03-09 19:10:03.785273 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.785277 | controller | - ' ' 2026-03-09 19:10:03.785282 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.785286 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.785290 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.785295 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.785299 | controller | --timeout=300s' 2026-03-09 19:10:03.785303 | controller | - ' ' 2026-03-09 19:10:03.785307 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.785312 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.785316 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.785320 | controller | - ' ' 2026-03-09 19:10:03.785324 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785329 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785333 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785337 | controller | - ' ' 2026-03-09 19:10:03.785341 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.785346 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.785350 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.785354 | controller | available"' 2026-03-09 19:10:03.785359 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.785363 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.785367 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.785371 | controller | - ' exit 1' 2026-03-09 19:10:03.785376 | controller | - ' fi' 2026-03-09 19:10:03.785380 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.785384 | controller | - ' ' 2026-03-09 19:10:03.785388 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.785393 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.785397 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785403 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.785407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785411 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.785416 | controller | -- \' 2026-03-09 19:10:03.785420 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785432 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.785436 | controller | - ' logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.785440 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.785445 | controller | - ' logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.785449 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.785453 | controller | found' 2026-03-09 19:10:03.785458 | controller | - ' logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.785462 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.785466 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.785470 | controller | - ' ' 2026-03-09 19:10:03.785475 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.785479 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.785483 | controller | - ' ' 2026-03-09 19:10:03.785488 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.785492 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.785496 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.785500 | controller | - ' exit 1' 2026-03-09 19:10:03.785505 | controller | - ' fi' 2026-03-09 19:10:03.785509 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.785513 | controller | - ' ' 2026-03-09 19:10:03.785518 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.785522 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.785526 | controller | | base64 -d)' 2026-03-09 19:10:03.785533 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.785537 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.785541 | controller | - ' exit 1' 2026-03-09 19:10:03.785546 | controller | - ' fi' 2026-03-09 19:10:03.785550 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.785554 | controller | - ' ' 2026-03-09 19:10:03.785558 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.785563 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.785569 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.785573 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.785577 | controller | --timeout=300s' 2026-03-09 19:10:03.785581 | controller | - ' ' 2026-03-09 19:10:03.785586 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.785590 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.785594 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.785599 | controller | - ' ' 2026-03-09 19:10:03.785605 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785609 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785613 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785617 | controller | - ' ' 2026-03-09 19:10:03.785622 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.785626 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.785630 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.785634 | controller | available"' 2026-03-09 19:10:03.785639 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.785644 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.785649 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.785653 | controller | - ' exit 1' 2026-03-09 19:10:03.785657 | controller | - ' fi' 2026-03-09 19:10:03.785662 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.785666 | controller | - ' ' 2026-03-09 19:10:03.785670 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.785674 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.785679 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785683 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.785687 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785691 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.785696 | controller | -- \' 2026-03-09 19:10:03.785701 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785706 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.785717 | controller | - ' logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.785721 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.785726 | controller | - ' logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.785730 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.785734 | controller | found' 2026-03-09 19:10:03.785739 | controller | - ' logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.785743 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.785747 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.785752 | controller | - ' ' 2026-03-09 19:10:03.785756 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.785760 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.785764 | controller | - ' ' 2026-03-09 19:10:03.785769 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.785773 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.785777 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.785781 | controller | - ' exit 1' 2026-03-09 19:10:03.785786 | controller | - ' fi' 2026-03-09 19:10:03.785790 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.785794 | controller | - ' ' 2026-03-09 19:10:03.785799 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.785805 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.785809 | controller | | base64 -d)' 2026-03-09 19:10:03.785813 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.785818 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.785822 | controller | - ' exit 1' 2026-03-09 19:10:03.785826 | controller | - ' fi' 2026-03-09 19:10:03.785830 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.785835 | controller | - ' ' 2026-03-09 19:10:03.785839 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.785843 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.785848 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.785852 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.785856 | controller | --timeout=300s' 2026-03-09 19:10:03.785860 | controller | - ' ' 2026-03-09 19:10:03.785865 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.785869 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.785873 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.785877 | controller | - ' ' 2026-03-09 19:10:03.785882 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785886 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785890 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.785894 | controller | - ' ' 2026-03-09 19:10:03.785899 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.785903 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.785907 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.785912 | controller | available"' 2026-03-09 19:10:03.785916 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.785920 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.785924 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.785929 | controller | - ' exit 1' 2026-03-09 19:10:03.785933 | controller | - ' fi' 2026-03-09 19:10:03.785937 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.785941 | controller | - ' ' 2026-03-09 19:10:03.785946 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.785950 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.785954 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785958 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.785963 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785967 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.785971 | controller | -- \' 2026-03-09 19:10:03.785976 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.785981 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.785993 | controller | - ' logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.785997 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.786002 | controller | - ' logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.786006 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.786010 | controller | found' 2026-03-09 19:10:03.786015 | controller | - ' logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.786019 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.786023 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.786028 | controller | - ' ' 2026-03-09 19:10:03.786032 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.786036 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.786041 | controller | - ' ' 2026-03-09 19:10:03.786045 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.786049 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.786053 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.786058 | controller | - ' exit 1' 2026-03-09 19:10:03.786062 | controller | - ' fi' 2026-03-09 19:10:03.786066 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.786070 | controller | - ' ' 2026-03-09 19:10:03.786075 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.786079 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.786083 | controller | | base64 -d)' 2026-03-09 19:10:03.786088 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.786092 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.786096 | controller | - ' exit 1' 2026-03-09 19:10:03.786100 | controller | - ' fi' 2026-03-09 19:10:03.786105 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.786109 | controller | - ' ' 2026-03-09 19:10:03.786113 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.786117 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.786122 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.786128 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.786132 | controller | --timeout=300s' 2026-03-09 19:10:03.786136 | controller | - ' ' 2026-03-09 19:10:03.786141 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.786159 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.786167 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.786172 | controller | - ' ' 2026-03-09 19:10:03.786176 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786180 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786185 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786189 | controller | - ' ' 2026-03-09 19:10:03.786193 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.786197 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.786203 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.786208 | controller | available"' 2026-03-09 19:10:03.786212 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.786216 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.786221 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.786225 | controller | - ' exit 1' 2026-03-09 19:10:03.786229 | controller | - ' fi' 2026-03-09 19:10:03.786233 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.786238 | controller | - ' ' 2026-03-09 19:10:03.786242 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.786246 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.786251 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786255 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.786259 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786263 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.786268 | controller | -- \' 2026-03-09 19:10:03.786272 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786276 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.786281 | controller | - ' logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.786285 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.786296 | controller | - ' logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.786301 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.786305 | controller | found' 2026-03-09 19:10:03.786310 | controller | - ' logger.go:42: 18:58:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.786314 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.786318 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.786323 | controller | - ' ' 2026-03-09 19:10:03.786327 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.786331 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.786336 | controller | - ' ' 2026-03-09 19:10:03.786340 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.786344 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.786348 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.786353 | controller | - ' exit 1' 2026-03-09 19:10:03.786357 | controller | - ' fi' 2026-03-09 19:10:03.786361 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.786366 | controller | - ' ' 2026-03-09 19:10:03.786370 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.786374 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.786378 | controller | | base64 -d)' 2026-03-09 19:10:03.786383 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.786387 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.786391 | controller | - ' exit 1' 2026-03-09 19:10:03.786396 | controller | - ' fi' 2026-03-09 19:10:03.786400 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.786406 | controller | - ' ' 2026-03-09 19:10:03.786410 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.786414 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.786419 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.786423 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.786427 | controller | --timeout=300s' 2026-03-09 19:10:03.786432 | controller | - ' ' 2026-03-09 19:10:03.786436 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.786440 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.786444 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.786449 | controller | - ' ' 2026-03-09 19:10:03.786453 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786457 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786462 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786466 | controller | - ' ' 2026-03-09 19:10:03.786470 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.786474 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.786479 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.786483 | controller | available"' 2026-03-09 19:10:03.786487 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.786492 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.786496 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.786500 | controller | - ' exit 1' 2026-03-09 19:10:03.786504 | controller | - ' fi' 2026-03-09 19:10:03.786509 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.786513 | controller | - ' ' 2026-03-09 19:10:03.786517 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.786522 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.786526 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786530 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.786535 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786539 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.786543 | controller | -- \' 2026-03-09 19:10:03.786548 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786552 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.786558 | controller | - ' logger.go:42: 18:58:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.786562 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.786573 | controller | - ' logger.go:42: 18:58:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.786577 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.786582 | controller | found' 2026-03-09 19:10:03.786588 | controller | - ' logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.786592 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.786596 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.786601 | controller | - ' ' 2026-03-09 19:10:03.786608 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.786612 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.786616 | controller | - ' ' 2026-03-09 19:10:03.786621 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.786625 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.786629 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.786634 | controller | - ' exit 1' 2026-03-09 19:10:03.786638 | controller | - ' fi' 2026-03-09 19:10:03.786642 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.786647 | controller | - ' ' 2026-03-09 19:10:03.786651 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.786655 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.786659 | controller | | base64 -d)' 2026-03-09 19:10:03.786664 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.786668 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.786672 | controller | - ' exit 1' 2026-03-09 19:10:03.786676 | controller | - ' fi' 2026-03-09 19:10:03.786681 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.786685 | controller | - ' ' 2026-03-09 19:10:03.786689 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.786694 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.786698 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.786702 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.786706 | controller | --timeout=300s' 2026-03-09 19:10:03.786712 | controller | - ' ' 2026-03-09 19:10:03.786717 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.786721 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.786725 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.786729 | controller | - ' ' 2026-03-09 19:10:03.786734 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786738 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786742 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.786747 | controller | - ' ' 2026-03-09 19:10:03.786751 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.786755 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.786759 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.786764 | controller | available"' 2026-03-09 19:10:03.786768 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.786772 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.786776 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.786781 | controller | - ' exit 1' 2026-03-09 19:10:03.786788 | controller | - ' fi' 2026-03-09 19:10:03.786792 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.786797 | controller | - ' ' 2026-03-09 19:10:03.786801 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.786805 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.786810 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786814 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.786818 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786822 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.786827 | controller | -- \' 2026-03-09 19:10:03.786831 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.786835 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.786839 | controller | - ' logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.786844 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.786848 | controller | - ' logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.786860 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.786864 | controller | found' 2026-03-09 19:10:03.786869 | controller | - ' logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.786873 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.786877 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.786882 | controller | - ' ' 2026-03-09 19:10:03.786886 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.786890 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.786894 | controller | - ' ' 2026-03-09 19:10:03.786899 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.786903 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.786907 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.786912 | controller | - ' exit 1' 2026-03-09 19:10:03.786916 | controller | - ' fi' 2026-03-09 19:10:03.786920 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.786925 | controller | - ' ' 2026-03-09 19:10:03.786931 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.786935 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.786939 | controller | | base64 -d)' 2026-03-09 19:10:03.786943 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.786948 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.786952 | controller | - ' exit 1' 2026-03-09 19:10:03.786956 | controller | - ' fi' 2026-03-09 19:10:03.786961 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.786965 | controller | - ' ' 2026-03-09 19:10:03.786969 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.786973 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.786978 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.786984 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.786988 | controller | --timeout=300s' 2026-03-09 19:10:03.786994 | controller | - ' ' 2026-03-09 19:10:03.786998 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.787003 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.787007 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.787011 | controller | - ' ' 2026-03-09 19:10:03.787015 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787020 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787024 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787028 | controller | - ' ' 2026-03-09 19:10:03.787033 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.787037 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.787041 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.787045 | controller | available"' 2026-03-09 19:10:03.787050 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.787054 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.787058 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.787063 | controller | - ' exit 1' 2026-03-09 19:10:03.787067 | controller | - ' fi' 2026-03-09 19:10:03.787071 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.787075 | controller | - ' ' 2026-03-09 19:10:03.787080 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.787084 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.787088 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787092 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.787098 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787103 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.787107 | controller | -- \' 2026-03-09 19:10:03.787111 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787115 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.787120 | controller | - ' logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.787124 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.787128 | controller | - ' logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.787139 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.787144 | controller | found' 2026-03-09 19:10:03.787160 | controller | - ' logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.787165 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.787169 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.787173 | controller | - ' ' 2026-03-09 19:10:03.787178 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.787182 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.787186 | controller | - ' ' 2026-03-09 19:10:03.787192 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.787197 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.787201 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.787205 | controller | - ' exit 1' 2026-03-09 19:10:03.787209 | controller | - ' fi' 2026-03-09 19:10:03.787214 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.787218 | controller | - ' ' 2026-03-09 19:10:03.787222 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.787227 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.787231 | controller | | base64 -d)' 2026-03-09 19:10:03.787235 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.787239 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.787244 | controller | - ' exit 1' 2026-03-09 19:10:03.787248 | controller | - ' fi' 2026-03-09 19:10:03.787252 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.787256 | controller | - ' ' 2026-03-09 19:10:03.787261 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.787265 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.787269 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.787274 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.787278 | controller | --timeout=300s' 2026-03-09 19:10:03.787282 | controller | - ' ' 2026-03-09 19:10:03.787287 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.787291 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.787295 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.787299 | controller | - ' ' 2026-03-09 19:10:03.787304 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787308 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787312 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787316 | controller | - ' ' 2026-03-09 19:10:03.787321 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.787325 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.787331 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.787335 | controller | available"' 2026-03-09 19:10:03.787340 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.787344 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.787348 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.787353 | controller | - ' exit 1' 2026-03-09 19:10:03.787357 | controller | - ' fi' 2026-03-09 19:10:03.787361 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.787365 | controller | - ' ' 2026-03-09 19:10:03.787370 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.787374 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.787378 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787382 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.787388 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787393 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.787397 | controller | -- \' 2026-03-09 19:10:03.787401 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787405 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.787410 | controller | - ' logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.787414 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.787418 | controller | - ' logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.787423 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.787427 | controller | found' 2026-03-09 19:10:03.787439 | controller | - ' logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.787443 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.787448 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.787452 | controller | - ' ' 2026-03-09 19:10:03.787456 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.787461 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.787465 | controller | - ' ' 2026-03-09 19:10:03.787469 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.787473 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.787478 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.787482 | controller | - ' exit 1' 2026-03-09 19:10:03.787486 | controller | - ' fi' 2026-03-09 19:10:03.787490 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.787495 | controller | - ' ' 2026-03-09 19:10:03.787499 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.787503 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.787507 | controller | | base64 -d)' 2026-03-09 19:10:03.787512 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.787516 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.787520 | controller | - ' exit 1' 2026-03-09 19:10:03.787524 | controller | - ' fi' 2026-03-09 19:10:03.787529 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.787533 | controller | - ' ' 2026-03-09 19:10:03.787537 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.787541 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.787546 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.787550 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.787554 | controller | --timeout=300s' 2026-03-09 19:10:03.787558 | controller | - ' ' 2026-03-09 19:10:03.787562 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.787567 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.787571 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.787575 | controller | - ' ' 2026-03-09 19:10:03.787580 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787586 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787590 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787594 | controller | - ' ' 2026-03-09 19:10:03.787600 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.787605 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.787609 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.787613 | controller | available"' 2026-03-09 19:10:03.787618 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.787622 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.787626 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.787630 | controller | - ' exit 1' 2026-03-09 19:10:03.787635 | controller | - ' fi' 2026-03-09 19:10:03.787639 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.787658 | controller | - ' ' 2026-03-09 19:10:03.787664 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.787668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.787672 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.787681 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787685 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.787689 | controller | -- \' 2026-03-09 19:10:03.787694 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787698 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.787702 | controller | - ' logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.787706 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.787711 | controller | - ' logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.787715 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.787719 | controller | found' 2026-03-09 19:10:03.787731 | controller | - ' logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.787735 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.787740 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.787744 | controller | - ' ' 2026-03-09 19:10:03.787748 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.787753 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.787757 | controller | - ' ' 2026-03-09 19:10:03.787761 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.787765 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.787770 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.787774 | controller | - ' exit 1' 2026-03-09 19:10:03.787778 | controller | - ' fi' 2026-03-09 19:10:03.787783 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.787787 | controller | - ' ' 2026-03-09 19:10:03.787791 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.787797 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.787802 | controller | | base64 -d)' 2026-03-09 19:10:03.787806 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.787810 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.787814 | controller | - ' exit 1' 2026-03-09 19:10:03.787819 | controller | - ' fi' 2026-03-09 19:10:03.787823 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.787827 | controller | - ' ' 2026-03-09 19:10:03.787831 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.787836 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.787840 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.787844 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.787848 | controller | --timeout=300s' 2026-03-09 19:10:03.787853 | controller | - ' ' 2026-03-09 19:10:03.787857 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.787861 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.787865 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.787870 | controller | - ' ' 2026-03-09 19:10:03.787876 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787880 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787885 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.787889 | controller | - ' ' 2026-03-09 19:10:03.787893 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.787897 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.787902 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.787906 | controller | available"' 2026-03-09 19:10:03.787910 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.787917 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.787922 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.787926 | controller | - ' exit 1' 2026-03-09 19:10:03.787930 | controller | - ' fi' 2026-03-09 19:10:03.787934 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.787939 | controller | - ' ' 2026-03-09 19:10:03.787943 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.787947 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.787952 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787956 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.787960 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787964 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.787969 | controller | -- \' 2026-03-09 19:10:03.787973 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.787977 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.787983 | controller | - ' logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.787987 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.787993 | controller | - ' logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.787998 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.788002 | controller | found' 2026-03-09 19:10:03.788006 | controller | - ' logger.go:42: 18:58:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.788011 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.788022 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.788027 | controller | - ' ' 2026-03-09 19:10:03.788031 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.788035 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.788040 | controller | - ' ' 2026-03-09 19:10:03.788044 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.788048 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.788052 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.788057 | controller | - ' exit 1' 2026-03-09 19:10:03.788061 | controller | - ' fi' 2026-03-09 19:10:03.788065 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.788069 | controller | - ' ' 2026-03-09 19:10:03.788074 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.788078 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.788082 | controller | | base64 -d)' 2026-03-09 19:10:03.788086 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.788091 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.788095 | controller | - ' exit 1' 2026-03-09 19:10:03.788099 | controller | - ' fi' 2026-03-09 19:10:03.788103 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.788108 | controller | - ' ' 2026-03-09 19:10:03.788112 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.788116 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.788120 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.788124 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.788129 | controller | --timeout=300s' 2026-03-09 19:10:03.788133 | controller | - ' ' 2026-03-09 19:10:03.788137 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.788141 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.788162 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.788168 | controller | - ' ' 2026-03-09 19:10:03.788172 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788179 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788183 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788187 | controller | - ' ' 2026-03-09 19:10:03.788192 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.788196 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.788202 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.788206 | controller | available"' 2026-03-09 19:10:03.788210 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.788216 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.788220 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.788225 | controller | - ' exit 1' 2026-03-09 19:10:03.788229 | controller | - ' fi' 2026-03-09 19:10:03.788233 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.788237 | controller | - ' ' 2026-03-09 19:10:03.788242 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.788246 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.788250 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788254 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.788259 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788263 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.788267 | controller | -- \' 2026-03-09 19:10:03.788271 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788276 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.788280 | controller | - ' logger.go:42: 18:58:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.788284 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.788288 | controller | - ' logger.go:42: 18:58:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.788293 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.788297 | controller | found' 2026-03-09 19:10:03.788301 | controller | - ' logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.788305 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.788310 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.788314 | controller | - ' ' 2026-03-09 19:10:03.788325 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.788330 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.788334 | controller | - ' ' 2026-03-09 19:10:03.788338 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.788343 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.788347 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.788351 | controller | - ' exit 1' 2026-03-09 19:10:03.788356 | controller | - ' fi' 2026-03-09 19:10:03.788360 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.788364 | controller | - ' ' 2026-03-09 19:10:03.788368 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.788373 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.788377 | controller | | base64 -d)' 2026-03-09 19:10:03.788381 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.788385 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.788390 | controller | - ' exit 1' 2026-03-09 19:10:03.788394 | controller | - ' fi' 2026-03-09 19:10:03.788398 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.788404 | controller | - ' ' 2026-03-09 19:10:03.788408 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.788413 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.788417 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.788421 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.788425 | controller | --timeout=300s' 2026-03-09 19:10:03.788430 | controller | - ' ' 2026-03-09 19:10:03.788434 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.788438 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.788443 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.788447 | controller | - ' ' 2026-03-09 19:10:03.788451 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788455 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788460 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788464 | controller | - ' ' 2026-03-09 19:10:03.788468 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.788472 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.788477 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.788481 | controller | available"' 2026-03-09 19:10:03.788485 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.788489 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.788494 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.788498 | controller | - ' exit 1' 2026-03-09 19:10:03.788502 | controller | - ' fi' 2026-03-09 19:10:03.788506 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.788511 | controller | - ' ' 2026-03-09 19:10:03.788516 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.788521 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.788525 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788529 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.788534 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788538 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.788542 | controller | -- \' 2026-03-09 19:10:03.788546 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788551 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.788555 | controller | - ' logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.788559 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.788564 | controller | - ' logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.788568 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.788572 | controller | found' 2026-03-09 19:10:03.788576 | controller | - ' logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.788582 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.788587 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.788591 | controller | - ' ' 2026-03-09 19:10:03.788602 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.788606 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.788610 | controller | - ' ' 2026-03-09 19:10:03.788615 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.788619 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.788624 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.788628 | controller | - ' exit 1' 2026-03-09 19:10:03.788632 | controller | - ' fi' 2026-03-09 19:10:03.788636 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.788641 | controller | - ' ' 2026-03-09 19:10:03.788645 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.788649 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.788654 | controller | | base64 -d)' 2026-03-09 19:10:03.788658 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.788662 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.788666 | controller | - ' exit 1' 2026-03-09 19:10:03.788671 | controller | - ' fi' 2026-03-09 19:10:03.788677 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.788681 | controller | - ' ' 2026-03-09 19:10:03.788685 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.788689 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.788694 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.788698 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.788702 | controller | --timeout=300s' 2026-03-09 19:10:03.788706 | controller | - ' ' 2026-03-09 19:10:03.788711 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.788715 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.788719 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.788723 | controller | - ' ' 2026-03-09 19:10:03.788728 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788732 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788736 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.788740 | controller | - ' ' 2026-03-09 19:10:03.788745 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.788749 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.788753 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.788757 | controller | available"' 2026-03-09 19:10:03.788762 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.788766 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.788770 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.788774 | controller | - ' exit 1' 2026-03-09 19:10:03.788779 | controller | - ' fi' 2026-03-09 19:10:03.788783 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.788789 | controller | - ' ' 2026-03-09 19:10:03.788793 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.788797 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.788802 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788806 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.788810 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788814 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.788819 | controller | -- \' 2026-03-09 19:10:03.788823 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.788827 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.788831 | controller | - ' logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.788836 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.788840 | controller | - ' logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.788844 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.788848 | controller | found' 2026-03-09 19:10:03.788853 | controller | - ' logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.788857 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.788861 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.788866 | controller | - ' ' 2026-03-09 19:10:03.788870 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.788880 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.788885 | controller | - ' ' 2026-03-09 19:10:03.788889 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.788893 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.788898 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.788902 | controller | - ' exit 1' 2026-03-09 19:10:03.788906 | controller | - ' fi' 2026-03-09 19:10:03.788911 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.788915 | controller | - ' ' 2026-03-09 19:10:03.788919 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.788923 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.788928 | controller | | base64 -d)' 2026-03-09 19:10:03.788932 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.788936 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.788940 | controller | - ' exit 1' 2026-03-09 19:10:03.788945 | controller | - ' fi' 2026-03-09 19:10:03.788949 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.788953 | controller | - ' ' 2026-03-09 19:10:03.788957 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.788962 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.788966 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.788970 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.788975 | controller | --timeout=300s' 2026-03-09 19:10:03.788979 | controller | - ' ' 2026-03-09 19:10:03.788983 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.788989 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.788993 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.788997 | controller | - ' ' 2026-03-09 19:10:03.789002 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789006 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789010 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789015 | controller | - ' ' 2026-03-09 19:10:03.789019 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.789023 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.789027 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.789031 | controller | available"' 2026-03-09 19:10:03.789036 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.789040 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.789044 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.789049 | controller | - ' exit 1' 2026-03-09 19:10:03.789053 | controller | - ' fi' 2026-03-09 19:10:03.789057 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.789061 | controller | - ' ' 2026-03-09 19:10:03.789066 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.789070 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.789074 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789080 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.789084 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789089 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.789094 | controller | -- \' 2026-03-09 19:10:03.789098 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789103 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.789107 | controller | - ' logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.789111 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.789116 | controller | - ' logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.789120 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.789124 | controller | found' 2026-03-09 19:10:03.789128 | controller | - ' logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.789133 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.789137 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.789141 | controller | - ' ' 2026-03-09 19:10:03.789156 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.789169 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.789174 | controller | - ' ' 2026-03-09 19:10:03.789178 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.789192 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.789196 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.789201 | controller | - ' exit 1' 2026-03-09 19:10:03.789205 | controller | - ' fi' 2026-03-09 19:10:03.789209 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.789214 | controller | - ' ' 2026-03-09 19:10:03.789218 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.789222 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.789226 | controller | | base64 -d)' 2026-03-09 19:10:03.789231 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.789238 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.789242 | controller | - ' exit 1' 2026-03-09 19:10:03.789247 | controller | - ' fi' 2026-03-09 19:10:03.789251 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.789255 | controller | - ' ' 2026-03-09 19:10:03.789260 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.789264 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.789268 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.789272 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.789277 | controller | --timeout=300s' 2026-03-09 19:10:03.789281 | controller | - ' ' 2026-03-09 19:10:03.789285 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.789289 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.789294 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.789298 | controller | - ' ' 2026-03-09 19:10:03.789302 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789306 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789311 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789315 | controller | - ' ' 2026-03-09 19:10:03.789319 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.789324 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.789328 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.789332 | controller | available"' 2026-03-09 19:10:03.789336 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.789342 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.789347 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.789351 | controller | - ' exit 1' 2026-03-09 19:10:03.789355 | controller | - ' fi' 2026-03-09 19:10:03.789359 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.789364 | controller | - ' ' 2026-03-09 19:10:03.789368 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.789372 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.789376 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789381 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.789387 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789391 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.789395 | controller | -- \' 2026-03-09 19:10:03.789400 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789404 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.789408 | controller | - ' logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.789412 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.789417 | controller | - ' logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.789421 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.789425 | controller | found' 2026-03-09 19:10:03.789430 | controller | - ' logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.789434 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.789438 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.789442 | controller | - ' ' 2026-03-09 19:10:03.789447 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.789451 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.789455 | controller | - ' ' 2026-03-09 19:10:03.789466 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.789471 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.789475 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.789479 | controller | - ' exit 1' 2026-03-09 19:10:03.789484 | controller | - ' fi' 2026-03-09 19:10:03.789488 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.789492 | controller | - ' ' 2026-03-09 19:10:03.789498 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.789502 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.789507 | controller | | base64 -d)' 2026-03-09 19:10:03.789511 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.789515 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.789520 | controller | - ' exit 1' 2026-03-09 19:10:03.789524 | controller | - ' fi' 2026-03-09 19:10:03.789528 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.789532 | controller | - ' ' 2026-03-09 19:10:03.789537 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.789541 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.789545 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.789549 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.789554 | controller | --timeout=300s' 2026-03-09 19:10:03.789558 | controller | - ' ' 2026-03-09 19:10:03.789562 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.789566 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.789571 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.789575 | controller | - ' ' 2026-03-09 19:10:03.789579 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789585 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789589 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789594 | controller | - ' ' 2026-03-09 19:10:03.789598 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.789602 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.789606 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.789611 | controller | available"' 2026-03-09 19:10:03.789615 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.789619 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.789623 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.789628 | controller | - ' exit 1' 2026-03-09 19:10:03.789632 | controller | - ' fi' 2026-03-09 19:10:03.789636 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.789641 | controller | - ' ' 2026-03-09 19:10:03.789645 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.789649 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.789653 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789657 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.789662 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789666 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.789670 | controller | -- \' 2026-03-09 19:10:03.789675 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789681 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.789685 | controller | - ' logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.789689 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.789694 | controller | - ' logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.789698 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.789702 | controller | found' 2026-03-09 19:10:03.789706 | controller | - ' logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.789711 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.789715 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.789719 | controller | - ' ' 2026-03-09 19:10:03.789723 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.789728 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.789732 | controller | - ' ' 2026-03-09 19:10:03.789743 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.789748 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.789752 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.789757 | controller | - ' exit 1' 2026-03-09 19:10:03.789763 | controller | - ' fi' 2026-03-09 19:10:03.789767 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.789772 | controller | - ' ' 2026-03-09 19:10:03.789776 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.789780 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.789786 | controller | | base64 -d)' 2026-03-09 19:10:03.789790 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.789794 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.789799 | controller | - ' exit 1' 2026-03-09 19:10:03.789803 | controller | - ' fi' 2026-03-09 19:10:03.789807 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.789812 | controller | - ' ' 2026-03-09 19:10:03.789816 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.789820 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.789839 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.789844 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.789848 | controller | --timeout=300s' 2026-03-09 19:10:03.789852 | controller | - ' ' 2026-03-09 19:10:03.789856 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.789861 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.789865 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.789869 | controller | - ' ' 2026-03-09 19:10:03.789873 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789878 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789882 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.789886 | controller | - ' ' 2026-03-09 19:10:03.789890 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.789895 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.789899 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.789903 | controller | available"' 2026-03-09 19:10:03.789908 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.789912 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.789916 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.789920 | controller | - ' exit 1' 2026-03-09 19:10:03.789925 | controller | - ' fi' 2026-03-09 19:10:03.789936 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.789941 | controller | - ' ' 2026-03-09 19:10:03.789945 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.789949 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.789953 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789958 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.789962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789966 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.789970 | controller | -- \' 2026-03-09 19:10:03.789975 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.789979 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.789985 | controller | - ' logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.789989 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.789993 | controller | - ' logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.789998 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.790002 | controller | found' 2026-03-09 19:10:03.790006 | controller | - ' logger.go:42: 18:58:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.790010 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.790015 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.790019 | controller | - ' ' 2026-03-09 19:10:03.790023 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.790028 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.790032 | controller | - ' ' 2026-03-09 19:10:03.790036 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.790040 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.790051 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.790056 | controller | - ' exit 1' 2026-03-09 19:10:03.790060 | controller | - ' fi' 2026-03-09 19:10:03.790064 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.790069 | controller | - ' ' 2026-03-09 19:10:03.790073 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.790077 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.790082 | controller | | base64 -d)' 2026-03-09 19:10:03.790086 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.790090 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.790094 | controller | - ' exit 1' 2026-03-09 19:10:03.790099 | controller | - ' fi' 2026-03-09 19:10:03.790103 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.790107 | controller | - ' ' 2026-03-09 19:10:03.790111 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.790116 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.790120 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.790124 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.790128 | controller | --timeout=300s' 2026-03-09 19:10:03.790133 | controller | - ' ' 2026-03-09 19:10:03.790137 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.790141 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.790155 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.790162 | controller | - ' ' 2026-03-09 19:10:03.790166 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790170 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790177 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790182 | controller | - ' ' 2026-03-09 19:10:03.790186 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.790190 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.790195 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.790201 | controller | available"' 2026-03-09 19:10:03.790205 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.790209 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.790213 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.790218 | controller | - ' exit 1' 2026-03-09 19:10:03.790222 | controller | - ' fi' 2026-03-09 19:10:03.790226 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.790231 | controller | - ' ' 2026-03-09 19:10:03.790235 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.790239 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.790243 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790247 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.790252 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790256 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.790260 | controller | -- \' 2026-03-09 19:10:03.790265 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790269 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.790273 | controller | - ' logger.go:42: 18:58:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.790277 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.790283 | controller | - ' logger.go:42: 18:58:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.790288 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.790292 | controller | found' 2026-03-09 19:10:03.790296 | controller | - ' logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.790300 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.790305 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.790309 | controller | - ' ' 2026-03-09 19:10:03.790313 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.790317 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.790322 | controller | - ' ' 2026-03-09 19:10:03.790326 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.790330 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.790342 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.790346 | controller | - ' exit 1' 2026-03-09 19:10:03.790351 | controller | - ' fi' 2026-03-09 19:10:03.790355 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.790359 | controller | - ' ' 2026-03-09 19:10:03.790372 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.790376 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.790381 | controller | | base64 -d)' 2026-03-09 19:10:03.790385 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.790389 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.790393 | controller | - ' exit 1' 2026-03-09 19:10:03.790398 | controller | - ' fi' 2026-03-09 19:10:03.790402 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.790406 | controller | - ' ' 2026-03-09 19:10:03.790410 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.790416 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.790421 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.790425 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.790429 | controller | --timeout=300s' 2026-03-09 19:10:03.790434 | controller | - ' ' 2026-03-09 19:10:03.790445 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.790449 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.790454 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.790458 | controller | - ' ' 2026-03-09 19:10:03.790462 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790466 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790471 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790475 | controller | - ' ' 2026-03-09 19:10:03.790479 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.790483 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.790488 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.790492 | controller | available"' 2026-03-09 19:10:03.790496 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.790500 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.790505 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.790509 | controller | - ' exit 1' 2026-03-09 19:10:03.790513 | controller | - ' fi' 2026-03-09 19:10:03.790518 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.790522 | controller | - ' ' 2026-03-09 19:10:03.790526 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.790530 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.790534 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790539 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.790543 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790547 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.790552 | controller | -- \' 2026-03-09 19:10:03.790556 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790560 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.790566 | controller | - ' logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.790570 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.790575 | controller | - ' logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.790579 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.790583 | controller | found' 2026-03-09 19:10:03.790587 | controller | - ' logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.790593 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.790598 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.790602 | controller | - ' ' 2026-03-09 19:10:03.790606 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.790610 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.790615 | controller | - ' ' 2026-03-09 19:10:03.790619 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.790623 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.790627 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.790632 | controller | - ' exit 1' 2026-03-09 19:10:03.790643 | controller | - ' fi' 2026-03-09 19:10:03.790647 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.790651 | controller | - ' ' 2026-03-09 19:10:03.790656 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.790660 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.790664 | controller | | base64 -d)' 2026-03-09 19:10:03.790669 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.790673 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.790677 | controller | - ' exit 1' 2026-03-09 19:10:03.790681 | controller | - ' fi' 2026-03-09 19:10:03.790686 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.790690 | controller | - ' ' 2026-03-09 19:10:03.790694 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.790698 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.790703 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.790707 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.790711 | controller | --timeout=300s' 2026-03-09 19:10:03.790718 | controller | - ' ' 2026-03-09 19:10:03.790723 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.790727 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.790731 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.790736 | controller | - ' ' 2026-03-09 19:10:03.790740 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790744 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790748 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.790752 | controller | - ' ' 2026-03-09 19:10:03.790757 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.790761 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.790765 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.790769 | controller | available"' 2026-03-09 19:10:03.790774 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.790778 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.790782 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.790787 | controller | - ' exit 1' 2026-03-09 19:10:03.790793 | controller | - ' fi' 2026-03-09 19:10:03.790798 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.790803 | controller | - ' ' 2026-03-09 19:10:03.790808 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.790812 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.790816 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790821 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.790825 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790829 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.790833 | controller | -- \' 2026-03-09 19:10:03.790838 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.790842 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.790846 | controller | - ' logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.790851 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.790855 | controller | - ' logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.790859 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.790863 | controller | found' 2026-03-09 19:10:03.790868 | controller | - ' logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.790872 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.790878 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.790882 | controller | - ' ' 2026-03-09 19:10:03.790886 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.790891 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.790895 | controller | - ' ' 2026-03-09 19:10:03.790899 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.790904 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.790908 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.790912 | controller | - ' exit 1' 2026-03-09 19:10:03.790916 | controller | - ' fi' 2026-03-09 19:10:03.790927 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.790932 | controller | - ' ' 2026-03-09 19:10:03.790936 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.790941 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.790945 | controller | | base64 -d)' 2026-03-09 19:10:03.790949 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.790953 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.790958 | controller | - ' exit 1' 2026-03-09 19:10:03.790962 | controller | - ' fi' 2026-03-09 19:10:03.790966 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.790970 | controller | - ' ' 2026-03-09 19:10:03.790975 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.790979 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.790983 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.790987 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.790992 | controller | --timeout=300s' 2026-03-09 19:10:03.790996 | controller | - ' ' 2026-03-09 19:10:03.791000 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.791006 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.791010 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.791015 | controller | - ' ' 2026-03-09 19:10:03.791019 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791023 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791028 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791032 | controller | - ' ' 2026-03-09 19:10:03.791036 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.791041 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.791045 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.791049 | controller | available"' 2026-03-09 19:10:03.791053 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.791058 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.791062 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.791066 | controller | - ' exit 1' 2026-03-09 19:10:03.791071 | controller | - ' fi' 2026-03-09 19:10:03.791075 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.791079 | controller | - ' ' 2026-03-09 19:10:03.791083 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.791088 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.791092 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791096 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.791101 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791106 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.791111 | controller | -- \' 2026-03-09 19:10:03.791116 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791121 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.791128 | controller | - ' logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.791133 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.791137 | controller | - ' logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.791142 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.791155 | controller | found' 2026-03-09 19:10:03.791162 | controller | - ' logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.791166 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.791170 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.791175 | controller | - ' ' 2026-03-09 19:10:03.791179 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.791183 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.791187 | controller | - ' ' 2026-03-09 19:10:03.791192 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.791196 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.791202 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.791206 | controller | - ' exit 1' 2026-03-09 19:10:03.791211 | controller | - ' fi' 2026-03-09 19:10:03.791215 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.791219 | controller | - ' ' 2026-03-09 19:10:03.791231 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.791236 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.791240 | controller | | base64 -d)' 2026-03-09 19:10:03.791245 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.791249 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.791253 | controller | - ' exit 1' 2026-03-09 19:10:03.791257 | controller | - ' fi' 2026-03-09 19:10:03.791262 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.791266 | controller | - ' ' 2026-03-09 19:10:03.791270 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.791275 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.791279 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.791283 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.791287 | controller | --timeout=300s' 2026-03-09 19:10:03.791292 | controller | - ' ' 2026-03-09 19:10:03.791296 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.791300 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.791304 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.791309 | controller | - ' ' 2026-03-09 19:10:03.791313 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791317 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791321 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791326 | controller | - ' ' 2026-03-09 19:10:03.791330 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.791334 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.791338 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.791343 | controller | available"' 2026-03-09 19:10:03.791347 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.791351 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.791356 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.791360 | controller | - ' exit 1' 2026-03-09 19:10:03.791364 | controller | - ' fi' 2026-03-09 19:10:03.791368 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.791373 | controller | - ' ' 2026-03-09 19:10:03.791377 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.791381 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.791385 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791390 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.791394 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791400 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.791404 | controller | -- \' 2026-03-09 19:10:03.791408 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791413 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.791417 | controller | - ' logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.791421 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.791426 | controller | - ' logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.791430 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.791434 | controller | found' 2026-03-09 19:10:03.791440 | controller | - ' logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.791445 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.791449 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.791455 | controller | - ' ' 2026-03-09 19:10:03.791459 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.791464 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.791468 | controller | - ' ' 2026-03-09 19:10:03.791472 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.791476 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.791481 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.791485 | controller | - ' exit 1' 2026-03-09 19:10:03.791490 | controller | - ' fi' 2026-03-09 19:10:03.791494 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.791498 | controller | - ' ' 2026-03-09 19:10:03.791503 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.791514 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.791518 | controller | | base64 -d)' 2026-03-09 19:10:03.791522 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.791527 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.791531 | controller | - ' exit 1' 2026-03-09 19:10:03.791535 | controller | - ' fi' 2026-03-09 19:10:03.791540 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.791544 | controller | - ' ' 2026-03-09 19:10:03.791548 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.791553 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.791557 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.791561 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.791565 | controller | --timeout=300s' 2026-03-09 19:10:03.791570 | controller | - ' ' 2026-03-09 19:10:03.791574 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.791578 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.791583 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.791587 | controller | - ' ' 2026-03-09 19:10:03.791591 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791595 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791601 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791606 | controller | - ' ' 2026-03-09 19:10:03.791610 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.791614 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.791619 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.791623 | controller | available"' 2026-03-09 19:10:03.791627 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.791632 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.791655 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.791661 | controller | - ' exit 1' 2026-03-09 19:10:03.791666 | controller | - ' fi' 2026-03-09 19:10:03.791670 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.791674 | controller | - ' ' 2026-03-09 19:10:03.791679 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.791683 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.791687 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791691 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.791696 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791700 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.791704 | controller | -- \' 2026-03-09 19:10:03.791709 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791713 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.791717 | controller | - ' logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.791723 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.791728 | controller | - ' logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.791732 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.791737 | controller | found' 2026-03-09 19:10:03.791741 | controller | - ' logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.791745 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.791749 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.791754 | controller | - ' ' 2026-03-09 19:10:03.791758 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.791762 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.791766 | controller | - ' ' 2026-03-09 19:10:03.791771 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.791775 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.791779 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.791783 | controller | - ' exit 1' 2026-03-09 19:10:03.791788 | controller | - ' fi' 2026-03-09 19:10:03.791792 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.791796 | controller | - ' ' 2026-03-09 19:10:03.791800 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.791812 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.791818 | controller | | base64 -d)' 2026-03-09 19:10:03.791823 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.791827 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.791831 | controller | - ' exit 1' 2026-03-09 19:10:03.791836 | controller | - ' fi' 2026-03-09 19:10:03.791840 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.791844 | controller | - ' ' 2026-03-09 19:10:03.791848 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.791853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.791857 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.791861 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.791866 | controller | --timeout=300s' 2026-03-09 19:10:03.791871 | controller | - ' ' 2026-03-09 19:10:03.791876 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.791880 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.791884 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.791889 | controller | - ' ' 2026-03-09 19:10:03.791896 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791900 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791904 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.791909 | controller | - ' ' 2026-03-09 19:10:03.791913 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.791917 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.791921 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.791926 | controller | available"' 2026-03-09 19:10:03.791930 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.791934 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.791938 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.791943 | controller | - ' exit 1' 2026-03-09 19:10:03.791947 | controller | - ' fi' 2026-03-09 19:10:03.791951 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.791955 | controller | - ' ' 2026-03-09 19:10:03.791960 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.791964 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.791968 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791972 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.791977 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791981 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.791985 | controller | -- \' 2026-03-09 19:10:03.791990 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.791994 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.791998 | controller | - ' logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.792004 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.792008 | controller | - ' logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.792013 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.792017 | controller | found' 2026-03-09 19:10:03.792021 | controller | - ' logger.go:42: 18:58:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.792026 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.792030 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.792034 | controller | - ' ' 2026-03-09 19:10:03.792040 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.792044 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.792048 | controller | - ' ' 2026-03-09 19:10:03.792053 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.792057 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.792061 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.792066 | controller | - ' exit 1' 2026-03-09 19:10:03.792070 | controller | - ' fi' 2026-03-09 19:10:03.792074 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.792078 | controller | - ' ' 2026-03-09 19:10:03.792084 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.792088 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.792099 | controller | | base64 -d)' 2026-03-09 19:10:03.792104 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.792108 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.792113 | controller | - ' exit 1' 2026-03-09 19:10:03.792117 | controller | - ' fi' 2026-03-09 19:10:03.792121 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.792125 | controller | - ' ' 2026-03-09 19:10:03.792130 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.792134 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.792138 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.792142 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.792159 | controller | --timeout=300s' 2026-03-09 19:10:03.792164 | controller | - ' ' 2026-03-09 19:10:03.792168 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.792172 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.792177 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.792181 | controller | - ' ' 2026-03-09 19:10:03.792185 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792189 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792194 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792198 | controller | - ' ' 2026-03-09 19:10:03.792202 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.792209 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.792214 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.792218 | controller | available"' 2026-03-09 19:10:03.792224 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.792228 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.792232 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.792237 | controller | - ' exit 1' 2026-03-09 19:10:03.792241 | controller | - ' fi' 2026-03-09 19:10:03.792245 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.792249 | controller | - ' ' 2026-03-09 19:10:03.792254 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.792258 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.792262 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792266 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.792271 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792275 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.792279 | controller | -- \' 2026-03-09 19:10:03.792284 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792288 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.792292 | controller | - ' logger.go:42: 18:58:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.792296 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.792301 | controller | - ' logger.go:42: 18:58:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.792305 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.792309 | controller | found' 2026-03-09 19:10:03.792313 | controller | - ' logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.792317 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.792322 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.792326 | controller | - ' ' 2026-03-09 19:10:03.792330 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.792335 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.792339 | controller | - ' ' 2026-03-09 19:10:03.792343 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.792347 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.792352 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.792356 | controller | - ' exit 1' 2026-03-09 19:10:03.792360 | controller | - ' fi' 2026-03-09 19:10:03.792364 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.792369 | controller | - ' ' 2026-03-09 19:10:03.792373 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.792377 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.792381 | controller | | base64 -d)' 2026-03-09 19:10:03.792393 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.792397 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.792402 | controller | - ' exit 1' 2026-03-09 19:10:03.792406 | controller | - ' fi' 2026-03-09 19:10:03.792410 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.792415 | controller | - ' ' 2026-03-09 19:10:03.792419 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.792423 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.792429 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.792433 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.792438 | controller | --timeout=300s' 2026-03-09 19:10:03.792442 | controller | - ' ' 2026-03-09 19:10:03.792446 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.792450 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.792455 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.792459 | controller | - ' ' 2026-03-09 19:10:03.792463 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792468 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792472 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792476 | controller | - ' ' 2026-03-09 19:10:03.792480 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.792485 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.792489 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.792493 | controller | available"' 2026-03-09 19:10:03.792497 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.792502 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.792506 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.792510 | controller | - ' exit 1' 2026-03-09 19:10:03.792514 | controller | - ' fi' 2026-03-09 19:10:03.792519 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.792523 | controller | - ' ' 2026-03-09 19:10:03.792527 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.792531 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.792536 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792540 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.792544 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792548 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.792553 | controller | -- \' 2026-03-09 19:10:03.792557 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792561 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.792566 | controller | - ' logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.792570 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.792574 | controller | - ' logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.792578 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.792583 | controller | found' 2026-03-09 19:10:03.792587 | controller | - ' logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.792591 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.792595 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.792602 | controller | - ' ' 2026-03-09 19:10:03.792606 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.792612 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.792616 | controller | - ' ' 2026-03-09 19:10:03.792620 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.792624 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.792629 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.792633 | controller | - ' exit 1' 2026-03-09 19:10:03.792637 | controller | - ' fi' 2026-03-09 19:10:03.792642 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.792646 | controller | - ' ' 2026-03-09 19:10:03.792650 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.792654 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.792659 | controller | | base64 -d)' 2026-03-09 19:10:03.792663 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.792674 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.792678 | controller | - ' exit 1' 2026-03-09 19:10:03.792682 | controller | - ' fi' 2026-03-09 19:10:03.792687 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.792691 | controller | - ' ' 2026-03-09 19:10:03.792695 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.792700 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.792704 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.792708 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.792713 | controller | --timeout=300s' 2026-03-09 19:10:03.792717 | controller | - ' ' 2026-03-09 19:10:03.792721 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.792725 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.792730 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.792734 | controller | - ' ' 2026-03-09 19:10:03.792738 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792742 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792747 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.792751 | controller | - ' ' 2026-03-09 19:10:03.792755 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.792759 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.792763 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.792768 | controller | available"' 2026-03-09 19:10:03.792772 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.792776 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.792780 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.792785 | controller | - ' exit 1' 2026-03-09 19:10:03.792789 | controller | - ' fi' 2026-03-09 19:10:03.792793 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.792797 | controller | - ' ' 2026-03-09 19:10:03.792803 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.792808 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.792812 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792816 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.792820 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792824 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.792829 | controller | -- \' 2026-03-09 19:10:03.792833 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.792837 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.792841 | controller | - ' logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.792846 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.792850 | controller | - ' logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.792854 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.792858 | controller | found' 2026-03-09 19:10:03.792863 | controller | - ' logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.792867 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.792871 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.792875 | controller | - ' ' 2026-03-09 19:10:03.792880 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.792884 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.792888 | controller | - ' ' 2026-03-09 19:10:03.792892 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.792897 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.792901 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.792905 | controller | - ' exit 1' 2026-03-09 19:10:03.792909 | controller | - ' fi' 2026-03-09 19:10:03.792914 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.792918 | controller | - ' ' 2026-03-09 19:10:03.792924 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.792928 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.792932 | controller | | base64 -d)' 2026-03-09 19:10:03.792937 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.792948 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.792952 | controller | - ' exit 1' 2026-03-09 19:10:03.792957 | controller | - ' fi' 2026-03-09 19:10:03.792961 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.792965 | controller | - ' ' 2026-03-09 19:10:03.792970 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.792974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.792978 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.792982 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.792987 | controller | --timeout=300s' 2026-03-09 19:10:03.792991 | controller | - ' ' 2026-03-09 19:10:03.792995 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.793000 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.793006 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.793010 | controller | - ' ' 2026-03-09 19:10:03.793014 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793018 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793023 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793027 | controller | - ' ' 2026-03-09 19:10:03.793031 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.793036 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.793040 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.793044 | controller | available"' 2026-03-09 19:10:03.793048 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.793053 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.793057 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.793061 | controller | - ' exit 1' 2026-03-09 19:10:03.793065 | controller | - ' fi' 2026-03-09 19:10:03.793070 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.793074 | controller | - ' ' 2026-03-09 19:10:03.793078 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.793082 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.793087 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793091 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.793095 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793099 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.793104 | controller | -- \' 2026-03-09 19:10:03.793108 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793112 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.793117 | controller | - ' logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.793121 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.793125 | controller | - ' logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.793129 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.793134 | controller | found' 2026-03-09 19:10:03.793138 | controller | - ' logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.793142 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.793158 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.793163 | controller | - ' ' 2026-03-09 19:10:03.793167 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.793171 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.793176 | controller | - ' ' 2026-03-09 19:10:03.793182 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.793186 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.793190 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.793196 | controller | - ' exit 1' 2026-03-09 19:10:03.793201 | controller | - ' fi' 2026-03-09 19:10:03.793205 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.793209 | controller | - ' ' 2026-03-09 19:10:03.793213 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.793218 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.793222 | controller | | base64 -d)' 2026-03-09 19:10:03.793226 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.793232 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.793236 | controller | - ' exit 1' 2026-03-09 19:10:03.793240 | controller | - ' fi' 2026-03-09 19:10:03.793252 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.793256 | controller | - ' ' 2026-03-09 19:10:03.793261 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.793265 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.793269 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.793274 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.793278 | controller | --timeout=300s' 2026-03-09 19:10:03.793282 | controller | - ' ' 2026-03-09 19:10:03.793287 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.793291 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.793298 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.793302 | controller | - ' ' 2026-03-09 19:10:03.793307 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793311 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793315 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793320 | controller | - ' ' 2026-03-09 19:10:03.793324 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.793328 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.793332 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.793336 | controller | available"' 2026-03-09 19:10:03.793341 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.793347 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.793352 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.793356 | controller | - ' exit 1' 2026-03-09 19:10:03.793360 | controller | - ' fi' 2026-03-09 19:10:03.793364 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.793369 | controller | - ' ' 2026-03-09 19:10:03.793373 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.793377 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.793381 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793386 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.793390 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793396 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.793400 | controller | -- \' 2026-03-09 19:10:03.793404 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793409 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.793413 | controller | - ' logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.793417 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.793421 | controller | - ' logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.793426 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.793430 | controller | found' 2026-03-09 19:10:03.793434 | controller | - ' logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.793438 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.793442 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.793447 | controller | - ' ' 2026-03-09 19:10:03.793451 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.793455 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.793459 | controller | - ' ' 2026-03-09 19:10:03.793464 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.793468 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.793472 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.793477 | controller | - ' exit 1' 2026-03-09 19:10:03.793481 | controller | - ' fi' 2026-03-09 19:10:03.793485 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.793489 | controller | - ' ' 2026-03-09 19:10:03.793494 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.793498 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.793502 | controller | | base64 -d)' 2026-03-09 19:10:03.793506 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.793511 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.793515 | controller | - ' exit 1' 2026-03-09 19:10:03.793519 | controller | - ' fi' 2026-03-09 19:10:03.793530 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.793534 | controller | - ' ' 2026-03-09 19:10:03.793539 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.793543 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.793547 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.793552 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.793556 | controller | --timeout=300s' 2026-03-09 19:10:03.793560 | controller | - ' ' 2026-03-09 19:10:03.793564 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.793569 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.793573 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.793577 | controller | - ' ' 2026-03-09 19:10:03.793582 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793586 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793590 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793596 | controller | - ' ' 2026-03-09 19:10:03.793600 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.793605 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.793609 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.793613 | controller | available"' 2026-03-09 19:10:03.793617 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.793622 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.793626 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.793630 | controller | - ' exit 1' 2026-03-09 19:10:03.793634 | controller | - ' fi' 2026-03-09 19:10:03.793639 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.793643 | controller | - ' ' 2026-03-09 19:10:03.793647 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.793651 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.793656 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793660 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.793664 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.793673 | controller | -- \' 2026-03-09 19:10:03.793677 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793681 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.793685 | controller | - ' logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.793690 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.793694 | controller | - ' logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.793698 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.793702 | controller | found' 2026-03-09 19:10:03.793707 | controller | - ' logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.793711 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.793715 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.793719 | controller | - ' ' 2026-03-09 19:10:03.793724 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.793728 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.793732 | controller | - ' ' 2026-03-09 19:10:03.793736 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.793741 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.793745 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.793749 | controller | - ' exit 1' 2026-03-09 19:10:03.793753 | controller | - ' fi' 2026-03-09 19:10:03.793758 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.793763 | controller | - ' ' 2026-03-09 19:10:03.793768 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.793772 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.793776 | controller | | base64 -d)' 2026-03-09 19:10:03.793781 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.793786 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.793791 | controller | - ' exit 1' 2026-03-09 19:10:03.793795 | controller | - ' fi' 2026-03-09 19:10:03.793799 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.793804 | controller | - ' ' 2026-03-09 19:10:03.793814 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.793819 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.793823 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.793828 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.793832 | controller | --timeout=300s' 2026-03-09 19:10:03.793836 | controller | - ' ' 2026-03-09 19:10:03.793841 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.793845 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.793849 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.793853 | controller | - ' ' 2026-03-09 19:10:03.793858 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793862 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793866 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.793870 | controller | - ' ' 2026-03-09 19:10:03.793875 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.793879 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.793883 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.793887 | controller | available"' 2026-03-09 19:10:03.793892 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.793896 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.793900 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.793905 | controller | - ' exit 1' 2026-03-09 19:10:03.793909 | controller | - ' fi' 2026-03-09 19:10:03.793913 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.793917 | controller | - ' ' 2026-03-09 19:10:03.793923 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.793928 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.793932 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793936 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.793940 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793945 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.793949 | controller | -- \' 2026-03-09 19:10:03.793953 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.793958 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.793962 | controller | - ' logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.793968 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.793972 | controller | - ' logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.793977 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.793981 | controller | found' 2026-03-09 19:10:03.793985 | controller | - ' logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.793989 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.793994 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.793998 | controller | - ' ' 2026-03-09 19:10:03.794002 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.794007 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.794011 | controller | - ' ' 2026-03-09 19:10:03.794015 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.794019 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.794024 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.794028 | controller | - ' exit 1' 2026-03-09 19:10:03.794032 | controller | - ' fi' 2026-03-09 19:10:03.794036 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.794041 | controller | - ' ' 2026-03-09 19:10:03.794045 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.794049 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.794053 | controller | | base64 -d)' 2026-03-09 19:10:03.794058 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.794062 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.794066 | controller | - ' exit 1' 2026-03-09 19:10:03.794070 | controller | - ' fi' 2026-03-09 19:10:03.794075 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.794079 | controller | - ' ' 2026-03-09 19:10:03.794083 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.794095 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.794100 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.794104 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.794109 | controller | --timeout=300s' 2026-03-09 19:10:03.794114 | controller | - ' ' 2026-03-09 19:10:03.794119 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.794124 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.794129 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.794134 | controller | - ' ' 2026-03-09 19:10:03.794139 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794143 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794159 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794164 | controller | - ' ' 2026-03-09 19:10:03.794168 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.794172 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.794177 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.794181 | controller | available"' 2026-03-09 19:10:03.794185 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.794191 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.794196 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.794200 | controller | - ' exit 1' 2026-03-09 19:10:03.794204 | controller | - ' fi' 2026-03-09 19:10:03.794209 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.794213 | controller | - ' ' 2026-03-09 19:10:03.794217 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.794221 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.794226 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794230 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.794236 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794240 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.794244 | controller | -- \' 2026-03-09 19:10:03.794249 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794253 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.794257 | controller | - ' logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.794262 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.794266 | controller | - ' logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.794270 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.794274 | controller | found' 2026-03-09 19:10:03.794279 | controller | - ' logger.go:42: 18:58:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.794283 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.794287 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.794292 | controller | - ' ' 2026-03-09 19:10:03.794296 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.794300 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.794304 | controller | - ' ' 2026-03-09 19:10:03.794309 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.794313 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.794317 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.794322 | controller | - ' exit 1' 2026-03-09 19:10:03.794326 | controller | - ' fi' 2026-03-09 19:10:03.794330 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.794334 | controller | - ' ' 2026-03-09 19:10:03.794339 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.794343 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.794347 | controller | | base64 -d)' 2026-03-09 19:10:03.794352 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.794356 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.794360 | controller | - ' exit 1' 2026-03-09 19:10:03.794364 | controller | - ' fi' 2026-03-09 19:10:03.794369 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.794373 | controller | - ' ' 2026-03-09 19:10:03.794377 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.794389 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.794421 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.794426 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.794431 | controller | --timeout=300s' 2026-03-09 19:10:03.794435 | controller | - ' ' 2026-03-09 19:10:03.794439 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.794444 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.794448 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.794452 | controller | - ' ' 2026-03-09 19:10:03.794457 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794461 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794465 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794469 | controller | - ' ' 2026-03-09 19:10:03.794473 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.794478 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.794482 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.794486 | controller | available"' 2026-03-09 19:10:03.794491 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.794495 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.794499 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.794503 | controller | - ' exit 1' 2026-03-09 19:10:03.794508 | controller | - ' fi' 2026-03-09 19:10:03.794512 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.794516 | controller | - ' ' 2026-03-09 19:10:03.794520 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.794525 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.794529 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794533 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.794538 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794542 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.794546 | controller | -- \' 2026-03-09 19:10:03.794551 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794555 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.794561 | controller | - ' logger.go:42: 18:58:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.794565 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.794569 | controller | - ' logger.go:42: 18:58:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.794574 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.794578 | controller | found' 2026-03-09 19:10:03.794582 | controller | - ' logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.794586 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.794591 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.794595 | controller | - ' ' 2026-03-09 19:10:03.794601 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.794605 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.794609 | controller | - ' ' 2026-03-09 19:10:03.794614 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.794618 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.794622 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.794626 | controller | - ' exit 1' 2026-03-09 19:10:03.794631 | controller | - ' fi' 2026-03-09 19:10:03.794635 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.794639 | controller | - ' ' 2026-03-09 19:10:03.794648 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.794653 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.794657 | controller | | base64 -d)' 2026-03-09 19:10:03.794661 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.794666 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.794670 | controller | - ' exit 1' 2026-03-09 19:10:03.794674 | controller | - ' fi' 2026-03-09 19:10:03.794678 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.794683 | controller | - ' ' 2026-03-09 19:10:03.794687 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.794691 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.794705 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.794710 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.794714 | controller | --timeout=300s' 2026-03-09 19:10:03.794719 | controller | - ' ' 2026-03-09 19:10:03.794723 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.794727 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.794732 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.794736 | controller | - ' ' 2026-03-09 19:10:03.794740 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794745 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794749 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.794753 | controller | - ' ' 2026-03-09 19:10:03.794757 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.794762 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.794766 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.794770 | controller | available"' 2026-03-09 19:10:03.794774 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.794779 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.794783 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.794787 | controller | - ' exit 1' 2026-03-09 19:10:03.794791 | controller | - ' fi' 2026-03-09 19:10:03.794796 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.794800 | controller | - ' ' 2026-03-09 19:10:03.794804 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.794810 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.794814 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794819 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.794823 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794827 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.794831 | controller | -- \' 2026-03-09 19:10:03.794836 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.794840 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.794844 | controller | - ' logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.794858 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.794863 | controller | - ' logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.794867 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.794872 | controller | found' 2026-03-09 19:10:03.794876 | controller | - ' logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.794880 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.794884 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.794889 | controller | - ' ' 2026-03-09 19:10:03.794893 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.794897 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.794901 | controller | - ' ' 2026-03-09 19:10:03.794906 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.794910 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.794914 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.794919 | controller | - ' exit 1' 2026-03-09 19:10:03.794923 | controller | - ' fi' 2026-03-09 19:10:03.794927 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.794932 | controller | - ' ' 2026-03-09 19:10:03.794936 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.794940 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.794944 | controller | | base64 -d)' 2026-03-09 19:10:03.794949 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.794953 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.794957 | controller | - ' exit 1' 2026-03-09 19:10:03.794961 | controller | - ' fi' 2026-03-09 19:10:03.794966 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.794970 | controller | - ' ' 2026-03-09 19:10:03.794974 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.794979 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.794991 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.794996 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.795000 | controller | --timeout=300s' 2026-03-09 19:10:03.795004 | controller | - ' ' 2026-03-09 19:10:03.795009 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.795013 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.795019 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.795023 | controller | - ' ' 2026-03-09 19:10:03.795030 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795034 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795038 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795042 | controller | - ' ' 2026-03-09 19:10:03.795047 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.795051 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.795055 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.795060 | controller | available"' 2026-03-09 19:10:03.795064 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.795068 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.795072 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.795077 | controller | - ' exit 1' 2026-03-09 19:10:03.795081 | controller | - ' fi' 2026-03-09 19:10:03.795085 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.795089 | controller | - ' ' 2026-03-09 19:10:03.795094 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.795098 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.795102 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795106 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.795111 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795115 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.795119 | controller | -- \' 2026-03-09 19:10:03.795123 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795128 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.795132 | controller | - ' logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.795136 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.795142 | controller | - ' logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.795159 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.795163 | controller | found' 2026-03-09 19:10:03.795168 | controller | - ' logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.795172 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.795176 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.795181 | controller | - ' ' 2026-03-09 19:10:03.795185 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.795189 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.795193 | controller | - ' ' 2026-03-09 19:10:03.795198 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.795202 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.795206 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.795213 | controller | - ' exit 1' 2026-03-09 19:10:03.795218 | controller | - ' fi' 2026-03-09 19:10:03.795222 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.795226 | controller | - ' ' 2026-03-09 19:10:03.795230 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.795235 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.795239 | controller | | base64 -d)' 2026-03-09 19:10:03.795243 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.795247 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.795252 | controller | - ' exit 1' 2026-03-09 19:10:03.795256 | controller | - ' fi' 2026-03-09 19:10:03.795260 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.795265 | controller | - ' ' 2026-03-09 19:10:03.795269 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.795273 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.795277 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.795289 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.795293 | controller | --timeout=300s' 2026-03-09 19:10:03.795298 | controller | - ' ' 2026-03-09 19:10:03.795302 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.795306 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.795311 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.795315 | controller | - ' ' 2026-03-09 19:10:03.795319 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795325 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795330 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795334 | controller | - ' ' 2026-03-09 19:10:03.795338 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.795342 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.795347 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.795351 | controller | available"' 2026-03-09 19:10:03.795355 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.795360 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.795364 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.795368 | controller | - ' exit 1' 2026-03-09 19:10:03.795372 | controller | - ' fi' 2026-03-09 19:10:03.795377 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.795381 | controller | - ' ' 2026-03-09 19:10:03.795385 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.795389 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.795394 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795398 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.795402 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795406 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.795413 | controller | -- \' 2026-03-09 19:10:03.795417 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795421 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.795426 | controller | - ' logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.795430 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.795434 | controller | - ' logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.795438 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.795443 | controller | found' 2026-03-09 19:10:03.795447 | controller | - ' logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.795451 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.795455 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.795460 | controller | - ' ' 2026-03-09 19:10:03.795464 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.795468 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.795472 | controller | - ' ' 2026-03-09 19:10:03.795477 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.795481 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.795485 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.795489 | controller | - ' exit 1' 2026-03-09 19:10:03.795494 | controller | - ' fi' 2026-03-09 19:10:03.795498 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.795502 | controller | - ' ' 2026-03-09 19:10:03.795507 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.795511 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.795515 | controller | | base64 -d)' 2026-03-09 19:10:03.795519 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.795524 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.795528 | controller | - ' exit 1' 2026-03-09 19:10:03.795532 | controller | - ' fi' 2026-03-09 19:10:03.795536 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.795541 | controller | - ' ' 2026-03-09 19:10:03.795545 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.795549 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.795553 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.795558 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.795570 | controller | --timeout=300s' 2026-03-09 19:10:03.795575 | controller | - ' ' 2026-03-09 19:10:03.795579 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.795584 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.795588 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.795592 | controller | - ' ' 2026-03-09 19:10:03.795597 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795601 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795605 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795611 | controller | - ' ' 2026-03-09 19:10:03.795615 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.795620 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.795624 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.795628 | controller | available"' 2026-03-09 19:10:03.795632 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.795657 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.795664 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.795668 | controller | - ' exit 1' 2026-03-09 19:10:03.795673 | controller | - ' fi' 2026-03-09 19:10:03.795677 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.795681 | controller | - ' ' 2026-03-09 19:10:03.795685 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.795689 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.795694 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795698 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.795702 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795706 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.795711 | controller | -- \' 2026-03-09 19:10:03.795715 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795719 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.795723 | controller | - ' logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.795728 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.795732 | controller | - ' logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.795736 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.795740 | controller | found' 2026-03-09 19:10:03.795745 | controller | - ' logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.795749 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.795753 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.795757 | controller | - ' ' 2026-03-09 19:10:03.795762 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.795766 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.795770 | controller | - ' ' 2026-03-09 19:10:03.795775 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.795779 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.795783 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.795787 | controller | - ' exit 1' 2026-03-09 19:10:03.795792 | controller | - ' fi' 2026-03-09 19:10:03.795796 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.795800 | controller | - ' ' 2026-03-09 19:10:03.795804 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.795809 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.795813 | controller | | base64 -d)' 2026-03-09 19:10:03.795817 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.795823 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.795828 | controller | - ' exit 1' 2026-03-09 19:10:03.795832 | controller | - ' fi' 2026-03-09 19:10:03.795836 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.795840 | controller | - ' ' 2026-03-09 19:10:03.795845 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.795849 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.795853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.795857 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.795862 | controller | --timeout=300s' 2026-03-09 19:10:03.795866 | controller | - ' ' 2026-03-09 19:10:03.795877 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.795882 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.795886 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.795891 | controller | - ' ' 2026-03-09 19:10:03.795897 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795902 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795907 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.795911 | controller | - ' ' 2026-03-09 19:10:03.795915 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.795919 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.795923 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.795928 | controller | available"' 2026-03-09 19:10:03.795932 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.795938 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.795942 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.795946 | controller | - ' exit 1' 2026-03-09 19:10:03.795951 | controller | - ' fi' 2026-03-09 19:10:03.795955 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.795959 | controller | - ' ' 2026-03-09 19:10:03.795964 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.795968 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.795972 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.795976 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.795997 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796002 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.796006 | controller | -- \' 2026-03-09 19:10:03.796011 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796015 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.796019 | controller | - ' logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.796024 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.796031 | controller | - ' logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.796035 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.796040 | controller | found' 2026-03-09 19:10:03.796044 | controller | - ' logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.796048 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.796052 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.796057 | controller | - ' ' 2026-03-09 19:10:03.796061 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.796065 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.796069 | controller | - ' ' 2026-03-09 19:10:03.796074 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.796078 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.796082 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.796086 | controller | - ' exit 1' 2026-03-09 19:10:03.796091 | controller | - ' fi' 2026-03-09 19:10:03.796095 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.796099 | controller | - ' ' 2026-03-09 19:10:03.796104 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.796108 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.796112 | controller | | base64 -d)' 2026-03-09 19:10:03.796116 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.796121 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.796125 | controller | - ' exit 1' 2026-03-09 19:10:03.796131 | controller | - ' fi' 2026-03-09 19:10:03.796136 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.796140 | controller | - ' ' 2026-03-09 19:10:03.796161 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.796168 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.796173 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.796177 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.796181 | controller | --timeout=300s' 2026-03-09 19:10:03.796185 | controller | - ' ' 2026-03-09 19:10:03.796200 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.796204 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.796209 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.796213 | controller | - ' ' 2026-03-09 19:10:03.796217 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796222 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796226 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796230 | controller | - ' ' 2026-03-09 19:10:03.796234 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.796239 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.796243 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.796247 | controller | available"' 2026-03-09 19:10:03.796251 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.796256 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.796262 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.796267 | controller | - ' exit 1' 2026-03-09 19:10:03.796271 | controller | - ' fi' 2026-03-09 19:10:03.796275 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.796279 | controller | - ' ' 2026-03-09 19:10:03.796284 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.796288 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.796292 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796296 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.796301 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796305 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.796311 | controller | -- \' 2026-03-09 19:10:03.796315 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796319 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.796324 | controller | - ' logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.796328 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.796332 | controller | - ' logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.796336 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.796341 | controller | found' 2026-03-09 19:10:03.796345 | controller | - ' logger.go:42: 18:58:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.796349 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.796353 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.796358 | controller | - ' ' 2026-03-09 19:10:03.796362 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.796366 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.796370 | controller | - ' ' 2026-03-09 19:10:03.796375 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.796379 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.796383 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.796387 | controller | - ' exit 1' 2026-03-09 19:10:03.796392 | controller | - ' fi' 2026-03-09 19:10:03.796396 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.796400 | controller | - ' ' 2026-03-09 19:10:03.796404 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.796408 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.796413 | controller | | base64 -d)' 2026-03-09 19:10:03.796417 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.796421 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.796425 | controller | - ' exit 1' 2026-03-09 19:10:03.796430 | controller | - ' fi' 2026-03-09 19:10:03.796434 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.796438 | controller | - ' ' 2026-03-09 19:10:03.796442 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.796446 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.796451 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.796457 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.796461 | controller | --timeout=300s' 2026-03-09 19:10:03.796467 | controller | - ' ' 2026-03-09 19:10:03.796471 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.796482 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.796487 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.796491 | controller | - ' ' 2026-03-09 19:10:03.796496 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796500 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796504 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796508 | controller | - ' ' 2026-03-09 19:10:03.796513 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.796517 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.796521 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.796526 | controller | available"' 2026-03-09 19:10:03.796530 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.796534 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.796538 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.796543 | controller | - ' exit 1' 2026-03-09 19:10:03.796547 | controller | - ' fi' 2026-03-09 19:10:03.796551 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.796555 | controller | - ' ' 2026-03-09 19:10:03.796560 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.796564 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.796568 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796572 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.796577 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796581 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.796585 | controller | -- \' 2026-03-09 19:10:03.796590 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796594 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.796598 | controller | - ' logger.go:42: 18:58:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.796602 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.796607 | controller | - ' logger.go:42: 18:58:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.796611 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.796615 | controller | found' 2026-03-09 19:10:03.796619 | controller | - ' logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.796624 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.796628 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.796632 | controller | - ' ' 2026-03-09 19:10:03.796636 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.796643 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.796647 | controller | - ' ' 2026-03-09 19:10:03.796651 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.796656 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.796660 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.796664 | controller | - ' exit 1' 2026-03-09 19:10:03.796668 | controller | - ' fi' 2026-03-09 19:10:03.796673 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.796677 | controller | - ' ' 2026-03-09 19:10:03.796681 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.796686 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.796690 | controller | | base64 -d)' 2026-03-09 19:10:03.796694 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.796698 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.796703 | controller | - ' exit 1' 2026-03-09 19:10:03.796707 | controller | - ' fi' 2026-03-09 19:10:03.796711 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.796716 | controller | - ' ' 2026-03-09 19:10:03.796720 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.796724 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.796728 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.796733 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.796737 | controller | --timeout=300s' 2026-03-09 19:10:03.796741 | controller | - ' ' 2026-03-09 19:10:03.796746 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.796756 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.796760 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.796765 | controller | - ' ' 2026-03-09 19:10:03.796769 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796773 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796778 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.796782 | controller | - ' ' 2026-03-09 19:10:03.796786 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.796790 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.796795 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.796799 | controller | available"' 2026-03-09 19:10:03.796803 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.796809 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.796814 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.796818 | controller | - ' exit 1' 2026-03-09 19:10:03.796822 | controller | - ' fi' 2026-03-09 19:10:03.796826 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.796831 | controller | - ' ' 2026-03-09 19:10:03.796835 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.796839 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.796845 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796849 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.796854 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796858 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.796862 | controller | -- \' 2026-03-09 19:10:03.796866 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.796871 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.796875 | controller | - ' logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.796879 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.796883 | controller | - ' logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.796888 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.796892 | controller | found' 2026-03-09 19:10:03.796896 | controller | - ' logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.796900 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.796905 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.796909 | controller | - ' ' 2026-03-09 19:10:03.796913 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.796918 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.796922 | controller | - ' ' 2026-03-09 19:10:03.796926 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.796930 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.796935 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.796939 | controller | - ' exit 1' 2026-03-09 19:10:03.796943 | controller | - ' fi' 2026-03-09 19:10:03.796947 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.796952 | controller | - ' ' 2026-03-09 19:10:03.796956 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.796960 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.796964 | controller | | base64 -d)' 2026-03-09 19:10:03.796969 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.796973 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.796977 | controller | - ' exit 1' 2026-03-09 19:10:03.796981 | controller | - ' fi' 2026-03-09 19:10:03.796986 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.796990 | controller | - ' ' 2026-03-09 19:10:03.796994 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.796998 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.797003 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.797007 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.797011 | controller | --timeout=300s' 2026-03-09 19:10:03.797015 | controller | - ' ' 2026-03-09 19:10:03.797020 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.797024 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.797036 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.797042 | controller | - ' ' 2026-03-09 19:10:03.797046 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797051 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797055 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797059 | controller | - ' ' 2026-03-09 19:10:03.797063 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.797068 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.797072 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.797076 | controller | available"' 2026-03-09 19:10:03.797081 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.797085 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.797089 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.797093 | controller | - ' exit 1' 2026-03-09 19:10:03.797098 | controller | - ' fi' 2026-03-09 19:10:03.797102 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.797106 | controller | - ' ' 2026-03-09 19:10:03.797110 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.797115 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.797119 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797123 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.797128 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797132 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.797136 | controller | -- \' 2026-03-09 19:10:03.797141 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797155 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.797162 | controller | - ' logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.797166 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.797170 | controller | - ' logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.797174 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.797179 | controller | found' 2026-03-09 19:10:03.797183 | controller | - ' logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.797187 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.797192 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.797196 | controller | - ' ' 2026-03-09 19:10:03.797200 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.797204 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.797209 | controller | - ' ' 2026-03-09 19:10:03.797215 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.797220 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.797224 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.797228 | controller | - ' exit 1' 2026-03-09 19:10:03.797233 | controller | - ' fi' 2026-03-09 19:10:03.797239 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.797243 | controller | - ' ' 2026-03-09 19:10:03.797249 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.797253 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.797257 | controller | | base64 -d)' 2026-03-09 19:10:03.797262 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.797267 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.797271 | controller | - ' exit 1' 2026-03-09 19:10:03.797276 | controller | - ' fi' 2026-03-09 19:10:03.797280 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.797284 | controller | - ' ' 2026-03-09 19:10:03.797289 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.797293 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.797297 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.797302 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.797306 | controller | --timeout=300s' 2026-03-09 19:10:03.797310 | controller | - ' ' 2026-03-09 19:10:03.797314 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.797319 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.797331 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.797336 | controller | - ' ' 2026-03-09 19:10:03.797340 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797345 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797352 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797356 | controller | - ' ' 2026-03-09 19:10:03.797361 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.797365 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.797369 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.797374 | controller | available"' 2026-03-09 19:10:03.797378 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.797382 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.797386 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.797391 | controller | - ' exit 1' 2026-03-09 19:10:03.797395 | controller | - ' fi' 2026-03-09 19:10:03.797399 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.797403 | controller | - ' ' 2026-03-09 19:10:03.797409 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.797413 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.797418 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797422 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.797426 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797430 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.797436 | controller | -- \' 2026-03-09 19:10:03.797441 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797445 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.797449 | controller | - ' logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.797453 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.797458 | controller | - ' logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.797462 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.797466 | controller | found' 2026-03-09 19:10:03.797471 | controller | - ' logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.797475 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.797479 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.797483 | controller | - ' ' 2026-03-09 19:10:03.797488 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.797492 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.797496 | controller | - ' ' 2026-03-09 19:10:03.797501 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.797505 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.797509 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.797513 | controller | - ' exit 1' 2026-03-09 19:10:03.797518 | controller | - ' fi' 2026-03-09 19:10:03.797522 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.797526 | controller | - ' ' 2026-03-09 19:10:03.797530 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.797534 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.797539 | controller | | base64 -d)' 2026-03-09 19:10:03.797543 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.797547 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.797552 | controller | - ' exit 1' 2026-03-09 19:10:03.797556 | controller | - ' fi' 2026-03-09 19:10:03.797560 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.797564 | controller | - ' ' 2026-03-09 19:10:03.797569 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.797573 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.797577 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.797581 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.797586 | controller | --timeout=300s' 2026-03-09 19:10:03.797590 | controller | - ' ' 2026-03-09 19:10:03.797594 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.797598 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.797610 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.797614 | controller | - ' ' 2026-03-09 19:10:03.797619 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797623 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797627 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797632 | controller | - ' ' 2026-03-09 19:10:03.797638 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.797642 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.797646 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.797651 | controller | available"' 2026-03-09 19:10:03.797655 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.797659 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.797663 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.797668 | controller | - ' exit 1' 2026-03-09 19:10:03.797672 | controller | - ' fi' 2026-03-09 19:10:03.797676 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.797681 | controller | - ' ' 2026-03-09 19:10:03.797685 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.797689 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.797693 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797698 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.797702 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797706 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.797710 | controller | -- \' 2026-03-09 19:10:03.797715 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797719 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.797723 | controller | - ' logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.797727 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.797732 | controller | - ' logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.797736 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.797740 | controller | found' 2026-03-09 19:10:03.797744 | controller | - ' logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.797749 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.797753 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.797757 | controller | - ' ' 2026-03-09 19:10:03.797762 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.797766 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.797770 | controller | - ' ' 2026-03-09 19:10:03.797774 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.797779 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.797783 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.797787 | controller | - ' exit 1' 2026-03-09 19:10:03.797792 | controller | - ' fi' 2026-03-09 19:10:03.797796 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.797800 | controller | - ' ' 2026-03-09 19:10:03.797805 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.797809 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.797813 | controller | | base64 -d)' 2026-03-09 19:10:03.797817 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.797822 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.797827 | controller | - ' exit 1' 2026-03-09 19:10:03.797832 | controller | - ' fi' 2026-03-09 19:10:03.797836 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.797840 | controller | - ' ' 2026-03-09 19:10:03.797845 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.797849 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.797853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.797857 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.797862 | controller | --timeout=300s' 2026-03-09 19:10:03.797866 | controller | - ' ' 2026-03-09 19:10:03.797870 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.797874 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.797879 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.797883 | controller | - ' ' 2026-03-09 19:10:03.797894 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797898 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797903 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.797907 | controller | - ' ' 2026-03-09 19:10:03.797911 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.797916 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.797920 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.797924 | controller | available"' 2026-03-09 19:10:03.797928 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.797933 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.797937 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.797941 | controller | - ' exit 1' 2026-03-09 19:10:03.797945 | controller | - ' fi' 2026-03-09 19:10:03.797950 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.797954 | controller | - ' ' 2026-03-09 19:10:03.797958 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.797962 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.797967 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797973 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.797977 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797981 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.797985 | controller | -- \' 2026-03-09 19:10:03.797990 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.797994 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.797998 | controller | - ' logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.798003 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.798007 | controller | - ' logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.798013 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.798017 | controller | found' 2026-03-09 19:10:03.798021 | controller | - ' logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.798026 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.798030 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.798034 | controller | - ' ' 2026-03-09 19:10:03.798038 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.798043 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.798047 | controller | - ' ' 2026-03-09 19:10:03.798051 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.798055 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.798060 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.798064 | controller | - ' exit 1' 2026-03-09 19:10:03.798068 | controller | - ' fi' 2026-03-09 19:10:03.798073 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.798077 | controller | - ' ' 2026-03-09 19:10:03.798081 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.798085 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.798090 | controller | | base64 -d)' 2026-03-09 19:10:03.798094 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.798098 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.798102 | controller | - ' exit 1' 2026-03-09 19:10:03.798107 | controller | - ' fi' 2026-03-09 19:10:03.798111 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.798115 | controller | - ' ' 2026-03-09 19:10:03.798120 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.798124 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.798128 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.798132 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.798137 | controller | --timeout=300s' 2026-03-09 19:10:03.798141 | controller | - ' ' 2026-03-09 19:10:03.798155 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.798162 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.798166 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.798170 | controller | - ' ' 2026-03-09 19:10:03.798182 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798186 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798190 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798195 | controller | - ' ' 2026-03-09 19:10:03.798199 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.798203 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.798208 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.798212 | controller | available"' 2026-03-09 19:10:03.798218 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.798222 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.798228 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.798233 | controller | - ' exit 1' 2026-03-09 19:10:03.798237 | controller | - ' fi' 2026-03-09 19:10:03.798241 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.798245 | controller | - ' ' 2026-03-09 19:10:03.798250 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.798254 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.798258 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798262 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.798267 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798271 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.798275 | controller | -- \' 2026-03-09 19:10:03.798279 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798284 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.798289 | controller | - ' logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.798294 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.798298 | controller | - ' logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.798302 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.798307 | controller | found' 2026-03-09 19:10:03.798311 | controller | - ' logger.go:42: 18:59:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.798315 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.798319 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.798324 | controller | - ' ' 2026-03-09 19:10:03.798328 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.798332 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.798337 | controller | - ' ' 2026-03-09 19:10:03.798341 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.798345 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.798349 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.798354 | controller | - ' exit 1' 2026-03-09 19:10:03.798358 | controller | - ' fi' 2026-03-09 19:10:03.798362 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.798366 | controller | - ' ' 2026-03-09 19:10:03.798371 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.798375 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.798379 | controller | | base64 -d)' 2026-03-09 19:10:03.798383 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.798388 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.798392 | controller | - ' exit 1' 2026-03-09 19:10:03.798396 | controller | - ' fi' 2026-03-09 19:10:03.798400 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.798405 | controller | - ' ' 2026-03-09 19:10:03.798409 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.798413 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.798417 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.798423 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.798427 | controller | --timeout=300s' 2026-03-09 19:10:03.798432 | controller | - ' ' 2026-03-09 19:10:03.798436 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.798440 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.798445 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.798449 | controller | - ' ' 2026-03-09 19:10:03.798453 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798465 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798470 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798474 | controller | - ' ' 2026-03-09 19:10:03.798478 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.798482 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.798487 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.798491 | controller | available"' 2026-03-09 19:10:03.798495 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.798502 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.798506 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.798510 | controller | - ' exit 1' 2026-03-09 19:10:03.798514 | controller | - ' fi' 2026-03-09 19:10:03.798519 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.798523 | controller | - ' ' 2026-03-09 19:10:03.798527 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.798531 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.798536 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798540 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.798544 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798548 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.798553 | controller | -- \' 2026-03-09 19:10:03.798557 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798562 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.798568 | controller | - ' logger.go:42: 18:59:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.798572 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.798577 | controller | - ' logger.go:42: 18:59:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.798581 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.798585 | controller | found' 2026-03-09 19:10:03.798590 | controller | - ' logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.798594 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.798598 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.798604 | controller | - ' ' 2026-03-09 19:10:03.798608 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.798615 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.798620 | controller | - ' ' 2026-03-09 19:10:03.798624 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.798628 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.798633 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.798637 | controller | - ' exit 1' 2026-03-09 19:10:03.798642 | controller | - ' fi' 2026-03-09 19:10:03.798646 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.798650 | controller | - ' ' 2026-03-09 19:10:03.798655 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.798659 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.798663 | controller | | base64 -d)' 2026-03-09 19:10:03.798668 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.798672 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.798676 | controller | - ' exit 1' 2026-03-09 19:10:03.798681 | controller | - ' fi' 2026-03-09 19:10:03.798685 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.798689 | controller | - ' ' 2026-03-09 19:10:03.798693 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.798698 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.798702 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.798706 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.798711 | controller | --timeout=300s' 2026-03-09 19:10:03.798715 | controller | - ' ' 2026-03-09 19:10:03.798719 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.798724 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.798728 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.798732 | controller | - ' ' 2026-03-09 19:10:03.798736 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798748 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798753 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.798757 | controller | - ' ' 2026-03-09 19:10:03.798761 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.798766 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.798770 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.798774 | controller | available"' 2026-03-09 19:10:03.798779 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.798783 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.798787 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.798794 | controller | - ' exit 1' 2026-03-09 19:10:03.798798 | controller | - ' fi' 2026-03-09 19:10:03.798803 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.798807 | controller | - ' ' 2026-03-09 19:10:03.798811 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.798816 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.798821 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798826 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.798830 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798834 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.798839 | controller | -- \' 2026-03-09 19:10:03.798843 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.798847 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.798852 | controller | - ' logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.798856 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.798862 | controller | - ' logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.798866 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.798870 | controller | found' 2026-03-09 19:10:03.798875 | controller | - ' logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.798879 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.798883 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.798888 | controller | - ' ' 2026-03-09 19:10:03.798892 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.798896 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.798900 | controller | - ' ' 2026-03-09 19:10:03.798905 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.798909 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.798913 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.798917 | controller | - ' exit 1' 2026-03-09 19:10:03.798922 | controller | - ' fi' 2026-03-09 19:10:03.798926 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.798930 | controller | - ' ' 2026-03-09 19:10:03.798935 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.798939 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.798943 | controller | | base64 -d)' 2026-03-09 19:10:03.798947 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.798952 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.798956 | controller | - ' exit 1' 2026-03-09 19:10:03.798960 | controller | - ' fi' 2026-03-09 19:10:03.798964 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.798969 | controller | - ' ' 2026-03-09 19:10:03.798973 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.798977 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.798982 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.798986 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.798990 | controller | --timeout=300s' 2026-03-09 19:10:03.798994 | controller | - ' ' 2026-03-09 19:10:03.798999 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.799003 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.799007 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.799013 | controller | - ' ' 2026-03-09 19:10:03.799017 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799022 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799033 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799037 | controller | - ' ' 2026-03-09 19:10:03.799042 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.799046 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.799050 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.799055 | controller | available"' 2026-03-09 19:10:03.799059 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.799063 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.799067 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.799071 | controller | - ' exit 1' 2026-03-09 19:10:03.799076 | controller | - ' fi' 2026-03-09 19:10:03.799080 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.799084 | controller | - ' ' 2026-03-09 19:10:03.799089 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.799093 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.799097 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799101 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.799106 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799110 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.799114 | controller | -- \' 2026-03-09 19:10:03.799118 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799123 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.799127 | controller | - ' logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.799131 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.799137 | controller | - ' logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.799141 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.799155 | controller | found' 2026-03-09 19:10:03.799161 | controller | - ' logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.799166 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.799170 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.799174 | controller | - ' ' 2026-03-09 19:10:03.799179 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.799183 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.799187 | controller | - ' ' 2026-03-09 19:10:03.799191 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.799195 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.799200 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.799204 | controller | - ' exit 1' 2026-03-09 19:10:03.799208 | controller | - ' fi' 2026-03-09 19:10:03.799213 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.799219 | controller | - ' ' 2026-03-09 19:10:03.799223 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.799227 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.799232 | controller | | base64 -d)' 2026-03-09 19:10:03.799236 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.799240 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.799244 | controller | - ' exit 1' 2026-03-09 19:10:03.799249 | controller | - ' fi' 2026-03-09 19:10:03.799253 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.799257 | controller | - ' ' 2026-03-09 19:10:03.799263 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.799267 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.799272 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.799276 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.799280 | controller | --timeout=300s' 2026-03-09 19:10:03.799285 | controller | - ' ' 2026-03-09 19:10:03.799289 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.799293 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.799297 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.799302 | controller | - ' ' 2026-03-09 19:10:03.799306 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799310 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799322 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799326 | controller | - ' ' 2026-03-09 19:10:03.799330 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.799336 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.799341 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.799345 | controller | available"' 2026-03-09 19:10:03.799349 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.799354 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.799358 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.799362 | controller | - ' exit 1' 2026-03-09 19:10:03.799366 | controller | - ' fi' 2026-03-09 19:10:03.799371 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.799375 | controller | - ' ' 2026-03-09 19:10:03.799379 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.799383 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.799388 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799392 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.799396 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799400 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.799405 | controller | -- \' 2026-03-09 19:10:03.799409 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799415 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.799419 | controller | - ' logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.799423 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.799428 | controller | - ' logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.799432 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.799436 | controller | found' 2026-03-09 19:10:03.799440 | controller | - ' logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.799445 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.799449 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.799453 | controller | - ' ' 2026-03-09 19:10:03.799457 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.799462 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.799466 | controller | - ' ' 2026-03-09 19:10:03.799470 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.799474 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.799479 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.799483 | controller | - ' exit 1' 2026-03-09 19:10:03.799487 | controller | - ' fi' 2026-03-09 19:10:03.799491 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.799496 | controller | - ' ' 2026-03-09 19:10:03.799500 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.799504 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.799508 | controller | | base64 -d)' 2026-03-09 19:10:03.799513 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.799517 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.799521 | controller | - ' exit 1' 2026-03-09 19:10:03.799525 | controller | - ' fi' 2026-03-09 19:10:03.799530 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.799534 | controller | - ' ' 2026-03-09 19:10:03.799538 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.799542 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.799547 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.799551 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.799555 | controller | --timeout=300s' 2026-03-09 19:10:03.799559 | controller | - ' ' 2026-03-09 19:10:03.799564 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.799568 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.799572 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.799576 | controller | - ' ' 2026-03-09 19:10:03.799581 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799585 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799589 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799600 | controller | - ' ' 2026-03-09 19:10:03.799604 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.799610 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.799614 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.799619 | controller | available"' 2026-03-09 19:10:03.799623 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.799627 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.799650 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.799656 | controller | - ' exit 1' 2026-03-09 19:10:03.799661 | controller | - ' fi' 2026-03-09 19:10:03.799665 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.799669 | controller | - ' ' 2026-03-09 19:10:03.799673 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.799678 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.799682 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799686 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.799690 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799694 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.799699 | controller | -- \' 2026-03-09 19:10:03.799703 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799707 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.799712 | controller | - ' logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.799716 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.799720 | controller | - ' logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.799727 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.799731 | controller | found' 2026-03-09 19:10:03.799736 | controller | - ' logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.799740 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.799744 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.799748 | controller | - ' ' 2026-03-09 19:10:03.799753 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.799757 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.799761 | controller | - ' ' 2026-03-09 19:10:03.799765 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.799770 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.799774 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.799778 | controller | - ' exit 1' 2026-03-09 19:10:03.799782 | controller | - ' fi' 2026-03-09 19:10:03.799787 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.799791 | controller | - ' ' 2026-03-09 19:10:03.799797 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.799801 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.799805 | controller | | base64 -d)' 2026-03-09 19:10:03.799810 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.799814 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.799818 | controller | - ' exit 1' 2026-03-09 19:10:03.799824 | controller | - ' fi' 2026-03-09 19:10:03.799828 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.799833 | controller | - ' ' 2026-03-09 19:10:03.799837 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.799841 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.799846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.799850 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.799854 | controller | --timeout=300s' 2026-03-09 19:10:03.799858 | controller | - ' ' 2026-03-09 19:10:03.799863 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.799867 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.799871 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.799875 | controller | - ' ' 2026-03-09 19:10:03.799881 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799885 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799889 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.799894 | controller | - ' ' 2026-03-09 19:10:03.799906 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.799911 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.799915 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.799919 | controller | available"' 2026-03-09 19:10:03.799923 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.799928 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.799932 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.799936 | controller | - ' exit 1' 2026-03-09 19:10:03.799940 | controller | - ' fi' 2026-03-09 19:10:03.799945 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.799949 | controller | - ' ' 2026-03-09 19:10:03.799953 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.799957 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.799962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799966 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.799970 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799975 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.799979 | controller | -- \' 2026-03-09 19:10:03.799983 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.799987 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.799992 | controller | - ' logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.799996 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.800000 | controller | - ' logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.800004 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.800012 | controller | found' 2026-03-09 19:10:03.800016 | controller | - ' logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.800020 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.800024 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.800029 | controller | - ' ' 2026-03-09 19:10:03.800033 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.800037 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.800041 | controller | - ' ' 2026-03-09 19:10:03.800046 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.800050 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.800054 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.800058 | controller | - ' exit 1' 2026-03-09 19:10:03.800063 | controller | - ' fi' 2026-03-09 19:10:03.800067 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.800071 | controller | - ' ' 2026-03-09 19:10:03.800075 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.800080 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.800084 | controller | | base64 -d)' 2026-03-09 19:10:03.800088 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.800093 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.800097 | controller | - ' exit 1' 2026-03-09 19:10:03.800101 | controller | - ' fi' 2026-03-09 19:10:03.800105 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.800110 | controller | - ' ' 2026-03-09 19:10:03.800114 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.800118 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.800122 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.800127 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.800131 | controller | --timeout=300s' 2026-03-09 19:10:03.800135 | controller | - ' ' 2026-03-09 19:10:03.800140 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.800162 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.800169 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.800175 | controller | - ' ' 2026-03-09 19:10:03.800180 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800184 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800189 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800193 | controller | - ' ' 2026-03-09 19:10:03.800197 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.800209 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.800213 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.800218 | controller | available"' 2026-03-09 19:10:03.800222 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.800226 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.800231 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.800237 | controller | - ' exit 1' 2026-03-09 19:10:03.800241 | controller | - ' fi' 2026-03-09 19:10:03.800246 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.800250 | controller | - ' ' 2026-03-09 19:10:03.800254 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.800258 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.800263 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800267 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.800271 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800275 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.800280 | controller | -- \' 2026-03-09 19:10:03.800284 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800288 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.800292 | controller | - ' logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.800297 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.800301 | controller | - ' logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.800305 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.800309 | controller | found' 2026-03-09 19:10:03.800316 | controller | - ' logger.go:42: 18:59:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.800320 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.800324 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.800328 | controller | - ' ' 2026-03-09 19:10:03.800333 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.800337 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.800341 | controller | - ' ' 2026-03-09 19:10:03.800345 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.800350 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.800354 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.800358 | controller | - ' exit 1' 2026-03-09 19:10:03.800363 | controller | - ' fi' 2026-03-09 19:10:03.800368 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.800373 | controller | - ' ' 2026-03-09 19:10:03.800377 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.800381 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.800386 | controller | | base64 -d)' 2026-03-09 19:10:03.800390 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.800394 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.800398 | controller | - ' exit 1' 2026-03-09 19:10:03.800403 | controller | - ' fi' 2026-03-09 19:10:03.800407 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.800411 | controller | - ' ' 2026-03-09 19:10:03.800415 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.800420 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.800424 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.800428 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.800434 | controller | --timeout=300s' 2026-03-09 19:10:03.800453 | controller | - ' ' 2026-03-09 19:10:03.800458 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.800462 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.800476 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.800480 | controller | - ' ' 2026-03-09 19:10:03.800485 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800489 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800493 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800497 | controller | - ' ' 2026-03-09 19:10:03.800502 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.800513 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.800517 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.800522 | controller | available"' 2026-03-09 19:10:03.800526 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.800530 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.800535 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.800539 | controller | - ' exit 1' 2026-03-09 19:10:03.800543 | controller | - ' fi' 2026-03-09 19:10:03.800548 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.800552 | controller | - ' ' 2026-03-09 19:10:03.800556 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.800560 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.800565 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800569 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.800573 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.800582 | controller | -- \' 2026-03-09 19:10:03.800586 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800590 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.800594 | controller | - ' logger.go:42: 18:59:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.800599 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.800603 | controller | - ' logger.go:42: 18:59:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.800607 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.800611 | controller | found' 2026-03-09 19:10:03.800616 | controller | - ' logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.800620 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.800624 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.800629 | controller | - ' ' 2026-03-09 19:10:03.800633 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.800637 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.800643 | controller | - ' ' 2026-03-09 19:10:03.800647 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.800652 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.800656 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.800660 | controller | - ' exit 1' 2026-03-09 19:10:03.800664 | controller | - ' fi' 2026-03-09 19:10:03.800669 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.800673 | controller | - ' ' 2026-03-09 19:10:03.800677 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.800682 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.800686 | controller | | base64 -d)' 2026-03-09 19:10:03.800690 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.800694 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.800699 | controller | - ' exit 1' 2026-03-09 19:10:03.800703 | controller | - ' fi' 2026-03-09 19:10:03.800707 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.800711 | controller | - ' ' 2026-03-09 19:10:03.800716 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.800720 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.800724 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.800728 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.800733 | controller | --timeout=300s' 2026-03-09 19:10:03.800737 | controller | - ' ' 2026-03-09 19:10:03.800741 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.800746 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.800750 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.800754 | controller | - ' ' 2026-03-09 19:10:03.800758 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800763 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800767 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.800771 | controller | - ' ' 2026-03-09 19:10:03.800775 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.800780 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.800790 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.800794 | controller | available"' 2026-03-09 19:10:03.800799 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.800803 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.800807 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.800812 | controller | - ' exit 1' 2026-03-09 19:10:03.800816 | controller | - ' fi' 2026-03-09 19:10:03.800820 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.800824 | controller | - ' ' 2026-03-09 19:10:03.800829 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.800833 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.800837 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800843 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.800848 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800852 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.800856 | controller | -- \' 2026-03-09 19:10:03.800861 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.800865 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.800869 | controller | - ' logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.800873 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.800878 | controller | - ' logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.800882 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.800886 | controller | found' 2026-03-09 19:10:03.800891 | controller | - ' logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.800895 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.800899 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.800903 | controller | - ' ' 2026-03-09 19:10:03.800909 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.800914 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.800918 | controller | - ' ' 2026-03-09 19:10:03.800922 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.800926 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.800930 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.800935 | controller | - ' exit 1' 2026-03-09 19:10:03.800939 | controller | - ' fi' 2026-03-09 19:10:03.800943 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.800947 | controller | - ' ' 2026-03-09 19:10:03.800952 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.800956 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.800960 | controller | | base64 -d)' 2026-03-09 19:10:03.800965 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.800969 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.800973 | controller | - ' exit 1' 2026-03-09 19:10:03.800977 | controller | - ' fi' 2026-03-09 19:10:03.800981 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.800986 | controller | - ' ' 2026-03-09 19:10:03.800990 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.800994 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.800998 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.801003 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.801007 | controller | --timeout=300s' 2026-03-09 19:10:03.801011 | controller | - ' ' 2026-03-09 19:10:03.801015 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.801020 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.801024 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.801028 | controller | - ' ' 2026-03-09 19:10:03.801034 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801038 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801042 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801047 | controller | - ' ' 2026-03-09 19:10:03.801051 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.801055 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.801059 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.801070 | controller | available"' 2026-03-09 19:10:03.801075 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.801080 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.801085 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.801089 | controller | - ' exit 1' 2026-03-09 19:10:03.801093 | controller | - ' fi' 2026-03-09 19:10:03.801098 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.801102 | controller | - ' ' 2026-03-09 19:10:03.801106 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.801110 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.801115 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801119 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.801123 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801127 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.801132 | controller | -- \' 2026-03-09 19:10:03.801136 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801140 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.801144 | controller | - ' logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.801161 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.801166 | controller | - ' logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.801172 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.801176 | controller | found' 2026-03-09 19:10:03.801180 | controller | - ' logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.801185 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.801189 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.801193 | controller | - ' ' 2026-03-09 19:10:03.801197 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.801202 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.801206 | controller | - ' ' 2026-03-09 19:10:03.801210 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.801214 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.801219 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.801223 | controller | - ' exit 1' 2026-03-09 19:10:03.801227 | controller | - ' fi' 2026-03-09 19:10:03.801232 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.801236 | controller | - ' ' 2026-03-09 19:10:03.801242 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.801246 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.801251 | controller | | base64 -d)' 2026-03-09 19:10:03.801255 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.801261 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.801266 | controller | - ' exit 1' 2026-03-09 19:10:03.801270 | controller | - ' fi' 2026-03-09 19:10:03.801274 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.801278 | controller | - ' ' 2026-03-09 19:10:03.801283 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.801287 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.801291 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.801295 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.801300 | controller | --timeout=300s' 2026-03-09 19:10:03.801304 | controller | - ' ' 2026-03-09 19:10:03.801308 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.801312 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.801317 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.801321 | controller | - ' ' 2026-03-09 19:10:03.801336 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801341 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801345 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801349 | controller | - ' ' 2026-03-09 19:10:03.801353 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.801358 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.801362 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.801366 | controller | available"' 2026-03-09 19:10:03.801378 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.801382 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.801386 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.801405 | controller | - ' exit 1' 2026-03-09 19:10:03.801410 | controller | - ' fi' 2026-03-09 19:10:03.801416 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.801420 | controller | - ' ' 2026-03-09 19:10:03.801424 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.801429 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.801433 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801437 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.801444 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801449 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.801453 | controller | -- \' 2026-03-09 19:10:03.801457 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801463 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.801467 | controller | - ' logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.801472 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.801476 | controller | - ' logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.801480 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.801485 | controller | found' 2026-03-09 19:10:03.801489 | controller | - ' logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.801493 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.801497 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.801502 | controller | - ' ' 2026-03-09 19:10:03.801506 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.801512 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.801516 | controller | - ' ' 2026-03-09 19:10:03.801520 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.801525 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.801529 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.801533 | controller | - ' exit 1' 2026-03-09 19:10:03.801537 | controller | - ' fi' 2026-03-09 19:10:03.801542 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.801546 | controller | - ' ' 2026-03-09 19:10:03.801550 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.801554 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.801559 | controller | | base64 -d)' 2026-03-09 19:10:03.801563 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.801567 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.801571 | controller | - ' exit 1' 2026-03-09 19:10:03.801576 | controller | - ' fi' 2026-03-09 19:10:03.801580 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.801584 | controller | - ' ' 2026-03-09 19:10:03.801589 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.801593 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.801597 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.801601 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.801605 | controller | --timeout=300s' 2026-03-09 19:10:03.801611 | controller | - ' ' 2026-03-09 19:10:03.801617 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.801623 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.801629 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.801635 | controller | - ' ' 2026-03-09 19:10:03.801641 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801646 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801650 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801655 | controller | - ' ' 2026-03-09 19:10:03.801659 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.801666 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.801670 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.801674 | controller | available"' 2026-03-09 19:10:03.801678 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.801691 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.801695 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.801699 | controller | - ' exit 1' 2026-03-09 19:10:03.801704 | controller | - ' fi' 2026-03-09 19:10:03.801708 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.801712 | controller | - ' ' 2026-03-09 19:10:03.801717 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.801721 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.801725 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801729 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.801734 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801738 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.801742 | controller | -- \' 2026-03-09 19:10:03.801746 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.801751 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.801755 | controller | - ' logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.801759 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.801763 | controller | - ' logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.801768 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.801772 | controller | found' 2026-03-09 19:10:03.801776 | controller | - ' logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.801780 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.801785 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.801789 | controller | - ' ' 2026-03-09 19:10:03.801793 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.801798 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.801802 | controller | - ' ' 2026-03-09 19:10:03.801806 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.801810 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.801815 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.801828 | controller | - ' exit 1' 2026-03-09 19:10:03.801832 | controller | - ' fi' 2026-03-09 19:10:03.801836 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.801841 | controller | - ' ' 2026-03-09 19:10:03.801845 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.801849 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.801853 | controller | | base64 -d)' 2026-03-09 19:10:03.801858 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.801862 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.801866 | controller | - ' exit 1' 2026-03-09 19:10:03.801871 | controller | - ' fi' 2026-03-09 19:10:03.801877 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.801881 | controller | - ' ' 2026-03-09 19:10:03.801885 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.801890 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.801894 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.801898 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.801902 | controller | --timeout=300s' 2026-03-09 19:10:03.801907 | controller | - ' ' 2026-03-09 19:10:03.801911 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.801915 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.801920 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.801924 | controller | - ' ' 2026-03-09 19:10:03.801928 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801932 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801937 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.801941 | controller | - ' ' 2026-03-09 19:10:03.801945 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.801949 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.801954 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.801958 | controller | available"' 2026-03-09 19:10:03.801962 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.801967 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.801978 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.801982 | controller | - ' exit 1' 2026-03-09 19:10:03.801986 | controller | - ' fi' 2026-03-09 19:10:03.801992 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.801998 | controller | - ' ' 2026-03-09 19:10:03.802004 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.802010 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.802016 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802021 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.802025 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802030 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.802034 | controller | -- \' 2026-03-09 19:10:03.802038 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802043 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.802047 | controller | - ' logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.802051 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.802055 | controller | - ' logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.802060 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.802066 | controller | found' 2026-03-09 19:10:03.802070 | controller | - ' logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.802075 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.802079 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.802083 | controller | - ' ' 2026-03-09 19:10:03.802088 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.802092 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.802096 | controller | - ' ' 2026-03-09 19:10:03.802103 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.802107 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.802111 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.802116 | controller | - ' exit 1' 2026-03-09 19:10:03.802120 | controller | - ' fi' 2026-03-09 19:10:03.802124 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.802129 | controller | - ' ' 2026-03-09 19:10:03.802133 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.802137 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.802141 | controller | | base64 -d)' 2026-03-09 19:10:03.802158 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.802165 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.802169 | controller | - ' exit 1' 2026-03-09 19:10:03.802173 | controller | - ' fi' 2026-03-09 19:10:03.802177 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.802182 | controller | - ' ' 2026-03-09 19:10:03.802186 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.802190 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.802194 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.802199 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.802203 | controller | --timeout=300s' 2026-03-09 19:10:03.802207 | controller | - ' ' 2026-03-09 19:10:03.802212 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.802216 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.802220 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.802224 | controller | - ' ' 2026-03-09 19:10:03.802229 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802233 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802237 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802241 | controller | - ' ' 2026-03-09 19:10:03.802246 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.802250 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.802254 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.802259 | controller | available"' 2026-03-09 19:10:03.802263 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.802267 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.802279 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.802284 | controller | - ' exit 1' 2026-03-09 19:10:03.802290 | controller | - ' fi' 2026-03-09 19:10:03.802294 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.802299 | controller | - ' ' 2026-03-09 19:10:03.802303 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.802308 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.802312 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802316 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.802330 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802334 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.802339 | controller | -- \' 2026-03-09 19:10:03.802343 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802347 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.802352 | controller | - ' logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.802356 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.802363 | controller | - ' logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.802367 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.802371 | controller | found' 2026-03-09 19:10:03.802376 | controller | - ' logger.go:42: 18:59:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.802380 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.802384 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.802389 | controller | - ' ' 2026-03-09 19:10:03.802393 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.802397 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.802401 | controller | - ' ' 2026-03-09 19:10:03.802406 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.802410 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.802414 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.802419 | controller | - ' exit 1' 2026-03-09 19:10:03.802423 | controller | - ' fi' 2026-03-09 19:10:03.802427 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.802431 | controller | - ' ' 2026-03-09 19:10:03.802437 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.802441 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.802446 | controller | | base64 -d)' 2026-03-09 19:10:03.802450 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.802454 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.802458 | controller | - ' exit 1' 2026-03-09 19:10:03.802463 | controller | - ' fi' 2026-03-09 19:10:03.802467 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.802471 | controller | - ' ' 2026-03-09 19:10:03.802477 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.802484 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.802490 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.802496 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.802501 | controller | --timeout=300s' 2026-03-09 19:10:03.802508 | controller | - ' ' 2026-03-09 19:10:03.802514 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.802518 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.802523 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.802527 | controller | - ' ' 2026-03-09 19:10:03.802531 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802537 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802541 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802545 | controller | - ' ' 2026-03-09 19:10:03.802550 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.802554 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.802558 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.802563 | controller | available"' 2026-03-09 19:10:03.802567 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.802571 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.802575 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.802580 | controller | - ' exit 1' 2026-03-09 19:10:03.802592 | controller | - ' fi' 2026-03-09 19:10:03.802597 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.802601 | controller | - ' ' 2026-03-09 19:10:03.802605 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.802610 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.802614 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802618 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.802622 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802627 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.802631 | controller | -- \' 2026-03-09 19:10:03.802635 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802639 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.802645 | controller | - ' logger.go:42: 18:59:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.802650 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.802654 | controller | - ' logger.go:42: 18:59:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.802658 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.802663 | controller | found' 2026-03-09 19:10:03.802667 | controller | - ' logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.802671 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.802675 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.802680 | controller | - ' ' 2026-03-09 19:10:03.802684 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.802688 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.802693 | controller | - ' ' 2026-03-09 19:10:03.802698 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.802703 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.802707 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.802711 | controller | - ' exit 1' 2026-03-09 19:10:03.802716 | controller | - ' fi' 2026-03-09 19:10:03.802720 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.802727 | controller | - ' ' 2026-03-09 19:10:03.802731 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.802736 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.802740 | controller | | base64 -d)' 2026-03-09 19:10:03.802744 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.802748 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.802753 | controller | - ' exit 1' 2026-03-09 19:10:03.802757 | controller | - ' fi' 2026-03-09 19:10:03.802761 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.802766 | controller | - ' ' 2026-03-09 19:10:03.802770 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.802774 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.802778 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.802783 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.802787 | controller | --timeout=300s' 2026-03-09 19:10:03.802793 | controller | - ' ' 2026-03-09 19:10:03.802797 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.802801 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.802806 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.802810 | controller | - ' ' 2026-03-09 19:10:03.802814 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802818 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802823 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.802827 | controller | - ' ' 2026-03-09 19:10:03.802831 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.802835 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.802840 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.802844 | controller | available"' 2026-03-09 19:10:03.802848 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.802852 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.802857 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.802861 | controller | - ' exit 1' 2026-03-09 19:10:03.802865 | controller | - ' fi' 2026-03-09 19:10:03.802876 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.802880 | controller | - ' ' 2026-03-09 19:10:03.802885 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.802889 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.802893 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802899 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.802904 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802908 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.802912 | controller | -- \' 2026-03-09 19:10:03.802916 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.802921 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.802925 | controller | - ' logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.802929 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.802934 | controller | - ' logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.802938 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.802942 | controller | found' 2026-03-09 19:10:03.802946 | controller | - ' logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.802951 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.802955 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.802959 | controller | - ' ' 2026-03-09 19:10:03.802963 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.802968 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.802972 | controller | - ' ' 2026-03-09 19:10:03.802976 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.802980 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.802985 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.802989 | controller | - ' exit 1' 2026-03-09 19:10:03.802993 | controller | - ' fi' 2026-03-09 19:10:03.802998 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.803002 | controller | - ' ' 2026-03-09 19:10:03.803006 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.803010 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.803015 | controller | | base64 -d)' 2026-03-09 19:10:03.803019 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.803023 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.803028 | controller | - ' exit 1' 2026-03-09 19:10:03.803032 | controller | - ' fi' 2026-03-09 19:10:03.803036 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.803040 | controller | - ' ' 2026-03-09 19:10:03.803045 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.803049 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.803053 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.803057 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.803062 | controller | --timeout=300s' 2026-03-09 19:10:03.803066 | controller | - ' ' 2026-03-09 19:10:03.803070 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.803075 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.803079 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.803083 | controller | - ' ' 2026-03-09 19:10:03.803087 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803094 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803098 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803102 | controller | - ' ' 2026-03-09 19:10:03.803106 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.803111 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.803115 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.803119 | controller | available"' 2026-03-09 19:10:03.803123 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.803128 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.803132 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.803136 | controller | - ' exit 1' 2026-03-09 19:10:03.803140 | controller | - ' fi' 2026-03-09 19:10:03.803156 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.803169 | controller | - ' ' 2026-03-09 19:10:03.803174 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.803178 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.803183 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803187 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.803191 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803195 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.803200 | controller | -- \' 2026-03-09 19:10:03.803204 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803208 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.803213 | controller | - ' logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.803217 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.803221 | controller | - ' logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.803226 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.803230 | controller | found' 2026-03-09 19:10:03.803234 | controller | - ' logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.803238 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.803243 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.803247 | controller | - ' ' 2026-03-09 19:10:03.803251 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.803256 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.803260 | controller | - ' ' 2026-03-09 19:10:03.803264 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.803269 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.803273 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.803277 | controller | - ' exit 1' 2026-03-09 19:10:03.803282 | controller | - ' fi' 2026-03-09 19:10:03.803286 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.803290 | controller | - ' ' 2026-03-09 19:10:03.803294 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.803300 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.803305 | controller | | base64 -d)' 2026-03-09 19:10:03.803309 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.803313 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.803318 | controller | - ' exit 1' 2026-03-09 19:10:03.803322 | controller | - ' fi' 2026-03-09 19:10:03.803326 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.803331 | controller | - ' ' 2026-03-09 19:10:03.803335 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.803339 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.803345 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.803350 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.803354 | controller | --timeout=300s' 2026-03-09 19:10:03.803358 | controller | - ' ' 2026-03-09 19:10:03.803362 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.803367 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.803371 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.803375 | controller | - ' ' 2026-03-09 19:10:03.803379 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803384 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803388 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803392 | controller | - ' ' 2026-03-09 19:10:03.803396 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.803401 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.803405 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.803409 | controller | available"' 2026-03-09 19:10:03.803413 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.803418 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.803422 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.803426 | controller | - ' exit 1' 2026-03-09 19:10:03.803430 | controller | - ' fi' 2026-03-09 19:10:03.803435 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.803439 | controller | - ' ' 2026-03-09 19:10:03.803450 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.803454 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.803458 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803463 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.803467 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803471 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.803477 | controller | -- \' 2026-03-09 19:10:03.803482 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803486 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.803492 | controller | - ' logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.803496 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.803500 | controller | - ' logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.803505 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.803509 | controller | found' 2026-03-09 19:10:03.803513 | controller | - ' logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.803517 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.803522 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.803526 | controller | - ' ' 2026-03-09 19:10:03.803530 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.803534 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.803539 | controller | - ' ' 2026-03-09 19:10:03.803543 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.803547 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.803551 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.803556 | controller | - ' exit 1' 2026-03-09 19:10:03.803560 | controller | - ' fi' 2026-03-09 19:10:03.803564 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.803569 | controller | - ' ' 2026-03-09 19:10:03.803574 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.803579 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.803583 | controller | | base64 -d)' 2026-03-09 19:10:03.803589 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.803593 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.803597 | controller | - ' exit 1' 2026-03-09 19:10:03.803602 | controller | - ' fi' 2026-03-09 19:10:03.803606 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.803610 | controller | - ' ' 2026-03-09 19:10:03.803614 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.803619 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.803623 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.803645 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.803651 | controller | --timeout=300s' 2026-03-09 19:10:03.803655 | controller | - ' ' 2026-03-09 19:10:03.803660 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.803664 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.803668 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.803673 | controller | - ' ' 2026-03-09 19:10:03.803677 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803681 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803685 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803690 | controller | - ' ' 2026-03-09 19:10:03.803694 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.803698 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.803704 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.803709 | controller | available"' 2026-03-09 19:10:03.803713 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.803719 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.803723 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.803727 | controller | - ' exit 1' 2026-03-09 19:10:03.803732 | controller | - ' fi' 2026-03-09 19:10:03.803736 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.803740 | controller | - ' ' 2026-03-09 19:10:03.803745 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.803756 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.803760 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803765 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.803769 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803773 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.803778 | controller | -- \' 2026-03-09 19:10:03.803782 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.803786 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.803790 | controller | - ' logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.803794 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.803799 | controller | - ' logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.803803 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.803807 | controller | found' 2026-03-09 19:10:03.803811 | controller | - ' logger.go:42: 18:59:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.803816 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.803820 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.803824 | controller | - ' ' 2026-03-09 19:10:03.803829 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.803833 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.803837 | controller | - ' ' 2026-03-09 19:10:03.803841 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.803846 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.803850 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.803854 | controller | - ' exit 1' 2026-03-09 19:10:03.803858 | controller | - ' fi' 2026-03-09 19:10:03.803863 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.803867 | controller | - ' ' 2026-03-09 19:10:03.803871 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.803876 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.803880 | controller | | base64 -d)' 2026-03-09 19:10:03.803884 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.803888 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.803893 | controller | - ' exit 1' 2026-03-09 19:10:03.803897 | controller | - ' fi' 2026-03-09 19:10:03.803901 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.803907 | controller | - ' ' 2026-03-09 19:10:03.803911 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.803916 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.803920 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.803926 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.803930 | controller | --timeout=300s' 2026-03-09 19:10:03.803934 | controller | - ' ' 2026-03-09 19:10:03.803938 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.803943 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.803947 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.803951 | controller | - ' ' 2026-03-09 19:10:03.803957 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803961 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803965 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.803970 | controller | - ' ' 2026-03-09 19:10:03.803974 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.803978 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.803982 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.803987 | controller | available"' 2026-03-09 19:10:03.803991 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.803995 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.804000 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.804004 | controller | - ' exit 1' 2026-03-09 19:10:03.804008 | controller | - ' fi' 2026-03-09 19:10:03.804012 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.804017 | controller | - ' ' 2026-03-09 19:10:03.804021 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.804032 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.804036 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804041 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.804045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804049 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.804054 | controller | -- \' 2026-03-09 19:10:03.804058 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804062 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.804066 | controller | - ' logger.go:42: 18:59:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.804071 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.804075 | controller | - ' logger.go:42: 18:59:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.804079 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.804083 | controller | found' 2026-03-09 19:10:03.804088 | controller | - ' logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.804096 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.804100 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.804105 | controller | - ' ' 2026-03-09 19:10:03.804110 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.804115 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.804119 | controller | - ' ' 2026-03-09 19:10:03.804125 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.804130 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.804135 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.804139 | controller | - ' exit 1' 2026-03-09 19:10:03.804143 | controller | - ' fi' 2026-03-09 19:10:03.804160 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.804165 | controller | - ' ' 2026-03-09 19:10:03.804169 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.804173 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.804177 | controller | | base64 -d)' 2026-03-09 19:10:03.804182 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.804186 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.804190 | controller | - ' exit 1' 2026-03-09 19:10:03.804195 | controller | - ' fi' 2026-03-09 19:10:03.804199 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.804203 | controller | - ' ' 2026-03-09 19:10:03.804207 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.804212 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.804216 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.804220 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.804224 | controller | --timeout=300s' 2026-03-09 19:10:03.804229 | controller | - ' ' 2026-03-09 19:10:03.804233 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.804237 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.804241 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.804246 | controller | - ' ' 2026-03-09 19:10:03.804250 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804254 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804258 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804263 | controller | - ' ' 2026-03-09 19:10:03.804267 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.804271 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.804275 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.804280 | controller | available"' 2026-03-09 19:10:03.804284 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.804288 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.804292 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.804297 | controller | - ' exit 1' 2026-03-09 19:10:03.804301 | controller | - ' fi' 2026-03-09 19:10:03.804307 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.804311 | controller | - ' ' 2026-03-09 19:10:03.804315 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.804320 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.804332 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804336 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.804341 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804345 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.804349 | controller | -- \' 2026-03-09 19:10:03.804354 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804358 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.804362 | controller | - ' logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.804367 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.804371 | controller | - ' logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.804375 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.804379 | controller | found' 2026-03-09 19:10:03.804384 | controller | - ' logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.804388 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.804392 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.804397 | controller | - ' ' 2026-03-09 19:10:03.804401 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.804405 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.804409 | controller | - ' ' 2026-03-09 19:10:03.804414 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.804418 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.804422 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.804426 | controller | - ' exit 1' 2026-03-09 19:10:03.804431 | controller | - ' fi' 2026-03-09 19:10:03.804435 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.804439 | controller | - ' ' 2026-03-09 19:10:03.804444 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.804448 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.804452 | controller | | base64 -d)' 2026-03-09 19:10:03.804456 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.804461 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.804465 | controller | - ' exit 1' 2026-03-09 19:10:03.804469 | controller | - ' fi' 2026-03-09 19:10:03.804474 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.804478 | controller | - ' ' 2026-03-09 19:10:03.804482 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.804486 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.804491 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.804495 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.804499 | controller | --timeout=300s' 2026-03-09 19:10:03.804504 | controller | - ' ' 2026-03-09 19:10:03.804509 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.804514 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.804518 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.804522 | controller | - ' ' 2026-03-09 19:10:03.804527 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804531 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804537 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804541 | controller | - ' ' 2026-03-09 19:10:03.804547 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.804552 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.804556 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.804560 | controller | available"' 2026-03-09 19:10:03.804564 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.804569 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.804573 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.804577 | controller | - ' exit 1' 2026-03-09 19:10:03.804581 | controller | - ' fi' 2026-03-09 19:10:03.804586 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.804590 | controller | - ' ' 2026-03-09 19:10:03.804594 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.804599 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.804610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804615 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.804619 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804623 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.804628 | controller | -- \' 2026-03-09 19:10:03.804632 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.804636 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.804640 | controller | - ' logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.804645 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.804649 | controller | - ' logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.804653 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.804657 | controller | found' 2026-03-09 19:10:03.804662 | controller | - ' logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.804666 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.804670 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.804675 | controller | - ' ' 2026-03-09 19:10:03.804681 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.804685 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.804689 | controller | - ' ' 2026-03-09 19:10:03.804694 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.804699 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.804704 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.804708 | controller | - ' exit 1' 2026-03-09 19:10:03.804712 | controller | - ' fi' 2026-03-09 19:10:03.804717 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.804721 | controller | - ' ' 2026-03-09 19:10:03.804725 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.804729 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.804734 | controller | | base64 -d)' 2026-03-09 19:10:03.804738 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.804742 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.804746 | controller | - ' exit 1' 2026-03-09 19:10:03.804751 | controller | - ' fi' 2026-03-09 19:10:03.804755 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.804759 | controller | - ' ' 2026-03-09 19:10:03.804763 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.804768 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.804772 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.804776 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.804781 | controller | --timeout=300s' 2026-03-09 19:10:03.804785 | controller | - ' ' 2026-03-09 19:10:03.804789 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.804793 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.804798 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.804802 | controller | - ' ' 2026-03-09 19:10:03.804806 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804810 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804815 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.804819 | controller | - ' ' 2026-03-09 19:10:03.804960 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.804968 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.804972 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.804976 | controller | available"' 2026-03-09 19:10:03.804981 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.804985 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.804989 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.804994 | controller | - ' exit 1' 2026-03-09 19:10:03.805013 | controller | - ' fi' 2026-03-09 19:10:03.805018 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.805022 | controller | - ' ' 2026-03-09 19:10:03.805026 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.805031 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.805035 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805047 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.805054 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805058 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.805063 | controller | -- \' 2026-03-09 19:10:03.805067 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805071 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.805076 | controller | - ' logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.805080 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.805084 | controller | - ' logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.805089 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.805093 | controller | found' 2026-03-09 19:10:03.805097 | controller | - ' logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.805101 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.805106 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.805110 | controller | - ' ' 2026-03-09 19:10:03.805114 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.805119 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.805123 | controller | - ' ' 2026-03-09 19:10:03.805127 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.805131 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.805136 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.805140 | controller | - ' exit 1' 2026-03-09 19:10:03.805144 | controller | - ' fi' 2026-03-09 19:10:03.805162 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.805166 | controller | - ' ' 2026-03-09 19:10:03.805173 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.805178 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.805184 | controller | | base64 -d)' 2026-03-09 19:10:03.805189 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.805195 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.805201 | controller | - ' exit 1' 2026-03-09 19:10:03.805207 | controller | - ' fi' 2026-03-09 19:10:03.805213 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.805219 | controller | - ' ' 2026-03-09 19:10:03.805224 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.805230 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.805235 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.805241 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.805247 | controller | --timeout=300s' 2026-03-09 19:10:03.805253 | controller | - ' ' 2026-03-09 19:10:03.805259 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.805265 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.805270 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.805276 | controller | - ' ' 2026-03-09 19:10:03.805282 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805290 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805295 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805301 | controller | - ' ' 2026-03-09 19:10:03.805307 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.805313 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.805322 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.805327 | controller | available"' 2026-03-09 19:10:03.805333 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.805338 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.805345 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.805351 | controller | - ' exit 1' 2026-03-09 19:10:03.805357 | controller | - ' fi' 2026-03-09 19:10:03.805363 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.805369 | controller | - ' ' 2026-03-09 19:10:03.805374 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.805380 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.805385 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805404 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.805411 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.805419 | controller | -- \' 2026-03-09 19:10:03.805424 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805428 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.805432 | controller | - ' logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.805437 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.805441 | controller | - ' logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.805445 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.805450 | controller | found' 2026-03-09 19:10:03.805454 | controller | - ' logger.go:42: 18:59:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.805458 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.805462 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.805467 | controller | - ' ' 2026-03-09 19:10:03.805471 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.805475 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.805480 | controller | - ' ' 2026-03-09 19:10:03.805484 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.805488 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.805492 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.805497 | controller | - ' exit 1' 2026-03-09 19:10:03.805501 | controller | - ' fi' 2026-03-09 19:10:03.805505 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.805510 | controller | - ' ' 2026-03-09 19:10:03.805514 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.805518 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.805525 | controller | | base64 -d)' 2026-03-09 19:10:03.805529 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.805536 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.805540 | controller | - ' exit 1' 2026-03-09 19:10:03.805544 | controller | - ' fi' 2026-03-09 19:10:03.805549 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.805553 | controller | - ' ' 2026-03-09 19:10:03.805557 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.805562 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.805566 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.805570 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.805574 | controller | --timeout=300s' 2026-03-09 19:10:03.805579 | controller | - ' ' 2026-03-09 19:10:03.805583 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.805587 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.805592 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.805596 | controller | - ' ' 2026-03-09 19:10:03.805600 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805605 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805609 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805613 | controller | - ' ' 2026-03-09 19:10:03.805617 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.805622 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.805626 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.805630 | controller | available"' 2026-03-09 19:10:03.805634 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.805639 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.805643 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.805647 | controller | - ' exit 1' 2026-03-09 19:10:03.805652 | controller | - ' fi' 2026-03-09 19:10:03.805656 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.805660 | controller | - ' ' 2026-03-09 19:10:03.805664 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.805669 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.805673 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.805691 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805696 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.805700 | controller | -- \' 2026-03-09 19:10:03.805704 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.805709 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.805716 | controller | - ' logger.go:42: 18:59:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.805721 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.805725 | controller | - ' logger.go:42: 18:59:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.805730 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.805734 | controller | found' 2026-03-09 19:10:03.805738 | controller | - ' logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.805742 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.805747 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.805751 | controller | - ' ' 2026-03-09 19:10:03.805755 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.805760 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.805764 | controller | - ' ' 2026-03-09 19:10:03.805768 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.805773 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.805777 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.805781 | controller | - ' exit 1' 2026-03-09 19:10:03.805785 | controller | - ' fi' 2026-03-09 19:10:03.805790 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.805794 | controller | - ' ' 2026-03-09 19:10:03.805798 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.805803 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.805807 | controller | | base64 -d)' 2026-03-09 19:10:03.805811 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.805816 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.805820 | controller | - ' exit 1' 2026-03-09 19:10:03.805824 | controller | - ' fi' 2026-03-09 19:10:03.805828 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.805833 | controller | - ' ' 2026-03-09 19:10:03.805837 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.805841 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.805846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.805850 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.805854 | controller | --timeout=300s' 2026-03-09 19:10:03.805858 | controller | - ' ' 2026-03-09 19:10:03.805863 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.805867 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.805871 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.805876 | controller | - ' ' 2026-03-09 19:10:03.805880 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805884 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805888 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.805893 | controller | - ' ' 2026-03-09 19:10:03.805900 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.805904 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.805909 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.805915 | controller | available"' 2026-03-09 19:10:03.805919 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.805963 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.805970 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.805974 | controller | - ' exit 1' 2026-03-09 19:10:03.805978 | controller | - ' fi' 2026-03-09 19:10:03.805983 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.805987 | controller | - ' ' 2026-03-09 19:10:03.805991 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.805995 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.806000 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806004 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.806016 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806020 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.806025 | controller | -- \' 2026-03-09 19:10:03.806029 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806033 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.806038 | controller | - ' logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.806042 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.806046 | controller | - ' logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.806050 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.806055 | controller | found' 2026-03-09 19:10:03.806059 | controller | - ' logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.806063 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.806068 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.806080 | controller | - ' ' 2026-03-09 19:10:03.806084 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.806089 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.806093 | controller | - ' ' 2026-03-09 19:10:03.806097 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.806102 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.806106 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.806110 | controller | - ' exit 1' 2026-03-09 19:10:03.806115 | controller | - ' fi' 2026-03-09 19:10:03.806119 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.806124 | controller | - ' ' 2026-03-09 19:10:03.806128 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.806132 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.806137 | controller | | base64 -d)' 2026-03-09 19:10:03.806183 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.806221 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.806227 | controller | - ' exit 1' 2026-03-09 19:10:03.806232 | controller | - ' fi' 2026-03-09 19:10:03.806236 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.806240 | controller | - ' ' 2026-03-09 19:10:03.806254 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.806259 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.806263 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.806268 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.806272 | controller | --timeout=300s' 2026-03-09 19:10:03.806276 | controller | - ' ' 2026-03-09 19:10:03.806281 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.806294 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.806299 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.806303 | controller | - ' ' 2026-03-09 19:10:03.806307 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806312 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806316 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806320 | controller | - ' ' 2026-03-09 19:10:03.806324 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.806329 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.806333 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.806337 | controller | available"' 2026-03-09 19:10:03.806342 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.806346 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.806350 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.806355 | controller | - ' exit 1' 2026-03-09 19:10:03.806359 | controller | - ' fi' 2026-03-09 19:10:03.806364 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.806368 | controller | - ' ' 2026-03-09 19:10:03.806372 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.806377 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.806381 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806385 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.806390 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806402 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.806407 | controller | -- \' 2026-03-09 19:10:03.806412 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806416 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.806420 | controller | - ' logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.806425 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.806429 | controller | - ' logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.806434 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.806438 | controller | found' 2026-03-09 19:10:03.806442 | controller | - ' logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.806449 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.806454 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.806458 | controller | - ' ' 2026-03-09 19:10:03.806462 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.806467 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.806471 | controller | - ' ' 2026-03-09 19:10:03.806475 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.806480 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.806484 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.806488 | controller | - ' exit 1' 2026-03-09 19:10:03.806492 | controller | - ' fi' 2026-03-09 19:10:03.806497 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.806501 | controller | - ' ' 2026-03-09 19:10:03.806505 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.806509 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.806514 | controller | | base64 -d)' 2026-03-09 19:10:03.806518 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.806522 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.806526 | controller | - ' exit 1' 2026-03-09 19:10:03.806555 | controller | - ' fi' 2026-03-09 19:10:03.806560 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.806564 | controller | - ' ' 2026-03-09 19:10:03.806570 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.806576 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.806582 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.806588 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.806593 | controller | --timeout=300s' 2026-03-09 19:10:03.806599 | controller | - ' ' 2026-03-09 19:10:03.806604 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.806609 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.806615 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.806620 | controller | - ' ' 2026-03-09 19:10:03.806626 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806632 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806638 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806643 | controller | - ' ' 2026-03-09 19:10:03.806648 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.806653 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.806659 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.806665 | controller | available"' 2026-03-09 19:10:03.806670 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.806679 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.806685 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.806691 | controller | - ' exit 1' 2026-03-09 19:10:03.806697 | controller | - ' fi' 2026-03-09 19:10:03.806703 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.806720 | controller | - ' ' 2026-03-09 19:10:03.806729 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.806734 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.806740 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806745 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.806751 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806768 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.806774 | controller | -- \' 2026-03-09 19:10:03.806779 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.806783 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.806788 | controller | - ' logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.806792 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.806796 | controller | - ' logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.806800 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.806805 | controller | found' 2026-03-09 19:10:03.806809 | controller | - ' logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.806813 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.806817 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.806822 | controller | - ' ' 2026-03-09 19:10:03.806826 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.806830 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.806834 | controller | - ' ' 2026-03-09 19:10:03.806839 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.806843 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.806847 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.806852 | controller | - ' exit 1' 2026-03-09 19:10:03.806856 | controller | - ' fi' 2026-03-09 19:10:03.806860 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.806864 | controller | - ' ' 2026-03-09 19:10:03.806868 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.806873 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.806877 | controller | | base64 -d)' 2026-03-09 19:10:03.806881 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.806886 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.806890 | controller | - ' exit 1' 2026-03-09 19:10:03.806894 | controller | - ' fi' 2026-03-09 19:10:03.806898 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.806903 | controller | - ' ' 2026-03-09 19:10:03.806907 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.806911 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.806916 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.806920 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.806924 | controller | --timeout=300s' 2026-03-09 19:10:03.806928 | controller | - ' ' 2026-03-09 19:10:03.806933 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.806939 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.806943 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.806948 | controller | - ' ' 2026-03-09 19:10:03.806952 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806956 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806961 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.806965 | controller | - ' ' 2026-03-09 19:10:03.806969 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.806975 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.806980 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.806984 | controller | available"' 2026-03-09 19:10:03.806988 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.806993 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.806997 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.807001 | controller | - ' exit 1' 2026-03-09 19:10:03.807005 | controller | - ' fi' 2026-03-09 19:10:03.807010 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.807014 | controller | - ' ' 2026-03-09 19:10:03.807018 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.807023 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.807027 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807031 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.807035 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807040 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.807044 | controller | -- \' 2026-03-09 19:10:03.807058 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807062 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.807067 | controller | - ' logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.807071 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.807075 | controller | - ' logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.807080 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.807084 | controller | found' 2026-03-09 19:10:03.807088 | controller | - ' logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.807092 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.807097 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.807101 | controller | - ' ' 2026-03-09 19:10:03.807105 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.807110 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.807114 | controller | - ' ' 2026-03-09 19:10:03.807118 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.807124 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.807128 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.807133 | controller | - ' exit 1' 2026-03-09 19:10:03.807137 | controller | - ' fi' 2026-03-09 19:10:03.807141 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.807159 | controller | - ' ' 2026-03-09 19:10:03.807166 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.807171 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.807175 | controller | | base64 -d)' 2026-03-09 19:10:03.807179 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.807184 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.807188 | controller | - ' exit 1' 2026-03-09 19:10:03.807192 | controller | - ' fi' 2026-03-09 19:10:03.807199 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.807203 | controller | - ' ' 2026-03-09 19:10:03.807208 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.807212 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.807216 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.807221 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.807225 | controller | --timeout=300s' 2026-03-09 19:10:03.807232 | controller | - ' ' 2026-03-09 19:10:03.807236 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.807241 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.807245 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.807249 | controller | - ' ' 2026-03-09 19:10:03.807254 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807258 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807262 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807302 | controller | - ' ' 2026-03-09 19:10:03.807307 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.807312 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.807316 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.807320 | controller | available"' 2026-03-09 19:10:03.807325 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.807329 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.807333 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.807337 | controller | - ' exit 1' 2026-03-09 19:10:03.807342 | controller | - ' fi' 2026-03-09 19:10:03.807346 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.807350 | controller | - ' ' 2026-03-09 19:10:03.807355 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.807359 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.807363 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807368 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.807373 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807380 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.807384 | controller | -- \' 2026-03-09 19:10:03.807396 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807401 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.807405 | controller | - ' logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.807409 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.807416 | controller | - ' logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.807420 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.807425 | controller | found' 2026-03-09 19:10:03.807430 | controller | - ' logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.807434 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.807438 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.807443 | controller | - ' ' 2026-03-09 19:10:03.807447 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.807451 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.807456 | controller | - ' ' 2026-03-09 19:10:03.807460 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.807464 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.807468 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.807473 | controller | - ' exit 1' 2026-03-09 19:10:03.807477 | controller | - ' fi' 2026-03-09 19:10:03.807481 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.807486 | controller | - ' ' 2026-03-09 19:10:03.807490 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.807494 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.807498 | controller | | base64 -d)' 2026-03-09 19:10:03.807503 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.807507 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.807511 | controller | - ' exit 1' 2026-03-09 19:10:03.807515 | controller | - ' fi' 2026-03-09 19:10:03.807520 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.807524 | controller | - ' ' 2026-03-09 19:10:03.807528 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.807533 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.807537 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.807541 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.807545 | controller | --timeout=300s' 2026-03-09 19:10:03.807550 | controller | - ' ' 2026-03-09 19:10:03.807554 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.807558 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.807562 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.807567 | controller | - ' ' 2026-03-09 19:10:03.807571 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807575 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807581 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807586 | controller | - ' ' 2026-03-09 19:10:03.807590 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.807594 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.807599 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.807603 | controller | available"' 2026-03-09 19:10:03.807607 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.807611 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.807616 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.807620 | controller | - ' exit 1' 2026-03-09 19:10:03.807643 | controller | - ' fi' 2026-03-09 19:10:03.807647 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.807651 | controller | - ' ' 2026-03-09 19:10:03.807656 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.807660 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.807664 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.807673 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.807681 | controller | -- \' 2026-03-09 19:10:03.807685 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807697 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.807701 | controller | - ' logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.807706 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.807710 | controller | - ' logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.807714 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.807719 | controller | found' 2026-03-09 19:10:03.807723 | controller | - ' logger.go:42: 18:59:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.807727 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.807732 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.807736 | controller | - ' ' 2026-03-09 19:10:03.807740 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.807744 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.807749 | controller | - ' ' 2026-03-09 19:10:03.807753 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.807757 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.807761 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.807766 | controller | - ' exit 1' 2026-03-09 19:10:03.807770 | controller | - ' fi' 2026-03-09 19:10:03.807774 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.807779 | controller | - ' ' 2026-03-09 19:10:03.807783 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.807787 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.807795 | controller | | base64 -d)' 2026-03-09 19:10:03.807799 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.807803 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.807808 | controller | - ' exit 1' 2026-03-09 19:10:03.807812 | controller | - ' fi' 2026-03-09 19:10:03.807816 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.807820 | controller | - ' ' 2026-03-09 19:10:03.807825 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.807829 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.807833 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.807838 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.807842 | controller | --timeout=300s' 2026-03-09 19:10:03.807846 | controller | - ' ' 2026-03-09 19:10:03.807850 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.807855 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.807859 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.807863 | controller | - ' ' 2026-03-09 19:10:03.807867 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807872 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807876 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.807880 | controller | - ' ' 2026-03-09 19:10:03.807884 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.807889 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.807893 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.807897 | controller | available"' 2026-03-09 19:10:03.807901 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.807906 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.807910 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.807914 | controller | - ' exit 1' 2026-03-09 19:10:03.807918 | controller | - ' fi' 2026-03-09 19:10:03.807923 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.807927 | controller | - ' ' 2026-03-09 19:10:03.807931 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.807935 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.807940 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807944 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.807948 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807952 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.807957 | controller | -- \' 2026-03-09 19:10:03.807961 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.807974 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.807978 | controller | - ' logger.go:42: 18:59:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.807985 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.807989 | controller | - ' logger.go:42: 18:59:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.807994 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.807998 | controller | found' 2026-03-09 19:10:03.808002 | controller | - ' logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.808006 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.808011 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.808015 | controller | - ' ' 2026-03-09 19:10:03.808019 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.808024 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.808028 | controller | - ' ' 2026-03-09 19:10:03.808032 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.808036 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.808041 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.808045 | controller | - ' exit 1' 2026-03-09 19:10:03.808049 | controller | - ' fi' 2026-03-09 19:10:03.808054 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.808058 | controller | - ' ' 2026-03-09 19:10:03.808064 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.808068 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.808072 | controller | | base64 -d)' 2026-03-09 19:10:03.808077 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.808083 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.808088 | controller | - ' exit 1' 2026-03-09 19:10:03.808094 | controller | - ' fi' 2026-03-09 19:10:03.808098 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.808102 | controller | - ' ' 2026-03-09 19:10:03.808106 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.808111 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.808115 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.808119 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.808123 | controller | --timeout=300s' 2026-03-09 19:10:03.808128 | controller | - ' ' 2026-03-09 19:10:03.808132 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.808136 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.808143 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.808166 | controller | - ' ' 2026-03-09 19:10:03.808171 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808175 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808180 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808184 | controller | - ' ' 2026-03-09 19:10:03.808188 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.808192 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.808197 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.808203 | controller | available"' 2026-03-09 19:10:03.808207 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.808211 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.808216 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.808220 | controller | - ' exit 1' 2026-03-09 19:10:03.808224 | controller | - ' fi' 2026-03-09 19:10:03.808229 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.808233 | controller | - ' ' 2026-03-09 19:10:03.808237 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.808241 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.808246 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808250 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.808254 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808258 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.808263 | controller | -- \' 2026-03-09 19:10:03.808267 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808271 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.808283 | controller | - ' logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.808288 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.808292 | controller | - ' logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.808296 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.808301 | controller | found' 2026-03-09 19:10:03.808305 | controller | - ' logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.808309 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.808313 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.808318 | controller | - ' ' 2026-03-09 19:10:03.808322 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.808326 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.808330 | controller | - ' ' 2026-03-09 19:10:03.808335 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.808339 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.808343 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.808348 | controller | - ' exit 1' 2026-03-09 19:10:03.808354 | controller | - ' fi' 2026-03-09 19:10:03.808358 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.808362 | controller | - ' ' 2026-03-09 19:10:03.808367 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.808371 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.808375 | controller | | base64 -d)' 2026-03-09 19:10:03.808379 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.808384 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.808388 | controller | - ' exit 1' 2026-03-09 19:10:03.808392 | controller | - ' fi' 2026-03-09 19:10:03.808396 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.808401 | controller | - ' ' 2026-03-09 19:10:03.808405 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.808411 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.808415 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.808419 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.808424 | controller | --timeout=300s' 2026-03-09 19:10:03.808428 | controller | - ' ' 2026-03-09 19:10:03.808432 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.808437 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.808441 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.808445 | controller | - ' ' 2026-03-09 19:10:03.808451 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808455 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808459 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808464 | controller | - ' ' 2026-03-09 19:10:03.808468 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.808472 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.808477 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.808481 | controller | available"' 2026-03-09 19:10:03.808485 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.808489 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.808494 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.808498 | controller | - ' exit 1' 2026-03-09 19:10:03.808502 | controller | - ' fi' 2026-03-09 19:10:03.808506 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.808511 | controller | - ' ' 2026-03-09 19:10:03.808515 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.808519 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.808523 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808528 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.808532 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808536 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.808540 | controller | -- \' 2026-03-09 19:10:03.808545 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808549 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.808561 | controller | - ' logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.808566 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.808570 | controller | - ' logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.808575 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.808579 | controller | found' 2026-03-09 19:10:03.808583 | controller | - ' logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.808587 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.808593 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.808598 | controller | - ' ' 2026-03-09 19:10:03.808602 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.808606 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.808611 | controller | - ' ' 2026-03-09 19:10:03.808615 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.808619 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.808623 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.808628 | controller | - ' exit 1' 2026-03-09 19:10:03.808632 | controller | - ' fi' 2026-03-09 19:10:03.808636 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.808641 | controller | - ' ' 2026-03-09 19:10:03.808645 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.808649 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.808653 | controller | | base64 -d)' 2026-03-09 19:10:03.808658 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.808662 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.808666 | controller | - ' exit 1' 2026-03-09 19:10:03.808670 | controller | - ' fi' 2026-03-09 19:10:03.808675 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.808679 | controller | - ' ' 2026-03-09 19:10:03.808683 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.808688 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.808692 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.808696 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.808700 | controller | --timeout=300s' 2026-03-09 19:10:03.808705 | controller | - ' ' 2026-03-09 19:10:03.808709 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.808713 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.808717 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.808722 | controller | - ' ' 2026-03-09 19:10:03.808726 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808730 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808734 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.808739 | controller | - ' ' 2026-03-09 19:10:03.808743 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.808747 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.808752 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.808756 | controller | available"' 2026-03-09 19:10:03.808760 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.808764 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.808769 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.808773 | controller | - ' exit 1' 2026-03-09 19:10:03.808777 | controller | - ' fi' 2026-03-09 19:10:03.808812 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.808819 | controller | - ' ' 2026-03-09 19:10:03.808825 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.808830 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.808834 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808838 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.808843 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808847 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.808851 | controller | -- \' 2026-03-09 19:10:03.808855 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.808860 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.808885 | controller | - ' logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.808897 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.808901 | controller | - ' logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.808906 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.808910 | controller | found' 2026-03-09 19:10:03.808914 | controller | - ' logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.808919 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.808923 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.808927 | controller | - ' ' 2026-03-09 19:10:03.808931 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.808936 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.808940 | controller | - ' ' 2026-03-09 19:10:03.808944 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.808948 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.808953 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.808957 | controller | - ' exit 1' 2026-03-09 19:10:03.808961 | controller | - ' fi' 2026-03-09 19:10:03.808965 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.808970 | controller | - ' ' 2026-03-09 19:10:03.808974 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.808978 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.808983 | controller | | base64 -d)' 2026-03-09 19:10:03.808987 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.808991 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.808995 | controller | - ' exit 1' 2026-03-09 19:10:03.809000 | controller | - ' fi' 2026-03-09 19:10:03.809004 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.809008 | controller | - ' ' 2026-03-09 19:10:03.809013 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.809017 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.809021 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.809025 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.809030 | controller | --timeout=300s' 2026-03-09 19:10:03.809034 | controller | - ' ' 2026-03-09 19:10:03.809038 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.809045 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.809050 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.809054 | controller | - ' ' 2026-03-09 19:10:03.809058 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809062 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809067 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809071 | controller | - ' ' 2026-03-09 19:10:03.809075 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.809079 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.809084 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.809088 | controller | available"' 2026-03-09 19:10:03.809092 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.809096 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.809101 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.809105 | controller | - ' exit 1' 2026-03-09 19:10:03.809109 | controller | - ' fi' 2026-03-09 19:10:03.809113 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.809118 | controller | - ' ' 2026-03-09 19:10:03.809122 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.809126 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.809131 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809135 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.809141 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809156 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.809163 | controller | -- \' 2026-03-09 19:10:03.809167 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809171 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.809176 | controller | - ' logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.809180 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.809194 | controller | - ' logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.809200 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.809205 | controller | found' 2026-03-09 19:10:03.809210 | controller | - ' logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.809215 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.809219 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.809223 | controller | - ' ' 2026-03-09 19:10:03.809228 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.809232 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.809236 | controller | - ' ' 2026-03-09 19:10:03.809241 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.809245 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.809249 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.809255 | controller | - ' exit 1' 2026-03-09 19:10:03.809260 | controller | - ' fi' 2026-03-09 19:10:03.809264 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.809268 | controller | - ' ' 2026-03-09 19:10:03.809272 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.809277 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.809281 | controller | | base64 -d)' 2026-03-09 19:10:03.809285 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.809290 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.809294 | controller | - ' exit 1' 2026-03-09 19:10:03.809298 | controller | - ' fi' 2026-03-09 19:10:03.809302 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.809307 | controller | - ' ' 2026-03-09 19:10:03.809311 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.809315 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.809319 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.809324 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.809328 | controller | --timeout=300s' 2026-03-09 19:10:03.809332 | controller | - ' ' 2026-03-09 19:10:03.809337 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.809341 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.809345 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.809349 | controller | - ' ' 2026-03-09 19:10:03.809354 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809358 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809362 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809366 | controller | - ' ' 2026-03-09 19:10:03.809371 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.809375 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.809379 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.809384 | controller | available"' 2026-03-09 19:10:03.809388 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.809394 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.809398 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.809403 | controller | - ' exit 1' 2026-03-09 19:10:03.809408 | controller | - ' fi' 2026-03-09 19:10:03.809413 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.809417 | controller | - ' ' 2026-03-09 19:10:03.809421 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.809426 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.809430 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809434 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.809438 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809444 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.809449 | controller | -- \' 2026-03-09 19:10:03.809453 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809457 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.809462 | controller | - ' logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.809466 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.809470 | controller | - ' logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.809481 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.809486 | controller | found' 2026-03-09 19:10:03.809490 | controller | - ' logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.809495 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.809499 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.809503 | controller | - ' ' 2026-03-09 19:10:03.809508 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.809512 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.809516 | controller | - ' ' 2026-03-09 19:10:03.809520 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.809525 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.809529 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.809533 | controller | - ' exit 1' 2026-03-09 19:10:03.809537 | controller | - ' fi' 2026-03-09 19:10:03.809542 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.809546 | controller | - ' ' 2026-03-09 19:10:03.809550 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.809554 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.809559 | controller | | base64 -d)' 2026-03-09 19:10:03.809563 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.809567 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.809572 | controller | - ' exit 1' 2026-03-09 19:10:03.809576 | controller | - ' fi' 2026-03-09 19:10:03.809580 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.809584 | controller | - ' ' 2026-03-09 19:10:03.809589 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.809593 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.809597 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.809601 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.809606 | controller | --timeout=300s' 2026-03-09 19:10:03.809610 | controller | - ' ' 2026-03-09 19:10:03.809614 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.809618 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.809623 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.809627 | controller | - ' ' 2026-03-09 19:10:03.809631 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809635 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809641 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809646 | controller | - ' ' 2026-03-09 19:10:03.809650 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.809654 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.809658 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.809663 | controller | available"' 2026-03-09 19:10:03.809667 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.809671 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.809676 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.809680 | controller | - ' exit 1' 2026-03-09 19:10:03.809684 | controller | - ' fi' 2026-03-09 19:10:03.809688 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.809693 | controller | - ' ' 2026-03-09 19:10:03.809697 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.809701 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.809705 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809710 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.809714 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809718 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.809722 | controller | -- \' 2026-03-09 19:10:03.809727 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809731 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.809735 | controller | - ' logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.809740 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.809744 | controller | - ' logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.809754 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.809759 | controller | found' 2026-03-09 19:10:03.809765 | controller | - ' logger.go:42: 18:59:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.809769 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.809774 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.809778 | controller | - ' ' 2026-03-09 19:10:03.809782 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.809787 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.809791 | controller | - ' ' 2026-03-09 19:10:03.809795 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.809799 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.809804 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.809808 | controller | - ' exit 1' 2026-03-09 19:10:03.809812 | controller | - ' fi' 2026-03-09 19:10:03.809817 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.809821 | controller | - ' ' 2026-03-09 19:10:03.809825 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.809830 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.809834 | controller | | base64 -d)' 2026-03-09 19:10:03.809840 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.809846 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.809850 | controller | - ' exit 1' 2026-03-09 19:10:03.809854 | controller | - ' fi' 2026-03-09 19:10:03.809859 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.809863 | controller | - ' ' 2026-03-09 19:10:03.809867 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.809871 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.809876 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.809880 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.809884 | controller | --timeout=300s' 2026-03-09 19:10:03.809889 | controller | - ' ' 2026-03-09 19:10:03.809893 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.809897 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.809901 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.809906 | controller | - ' ' 2026-03-09 19:10:03.809910 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809914 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809918 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.809923 | controller | - ' ' 2026-03-09 19:10:03.809927 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.809931 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.809935 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.809940 | controller | available"' 2026-03-09 19:10:03.809944 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.809948 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.809952 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.809957 | controller | - ' exit 1' 2026-03-09 19:10:03.809961 | controller | - ' fi' 2026-03-09 19:10:03.809965 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.809969 | controller | - ' ' 2026-03-09 19:10:03.809974 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.809978 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.809982 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809986 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.809991 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.809995 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.809999 | controller | -- \' 2026-03-09 19:10:03.810004 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810008 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.810012 | controller | - ' logger.go:42: 18:59:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.810018 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.810024 | controller | - ' logger.go:42: 18:59:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.810028 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.810033 | controller | found' 2026-03-09 19:10:03.810044 | controller | - ' logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.810049 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.810053 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.810058 | controller | - ' ' 2026-03-09 19:10:03.810062 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.810066 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.810071 | controller | - ' ' 2026-03-09 19:10:03.810075 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.810079 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.810083 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.810088 | controller | - ' exit 1' 2026-03-09 19:10:03.810092 | controller | - ' fi' 2026-03-09 19:10:03.810096 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.810100 | controller | - ' ' 2026-03-09 19:10:03.810105 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.810109 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.810113 | controller | | base64 -d)' 2026-03-09 19:10:03.810117 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.810122 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.810126 | controller | - ' exit 1' 2026-03-09 19:10:03.810130 | controller | - ' fi' 2026-03-09 19:10:03.810134 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.810139 | controller | - ' ' 2026-03-09 19:10:03.810143 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.810159 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.810164 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.810168 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.810172 | controller | --timeout=300s' 2026-03-09 19:10:03.810177 | controller | - ' ' 2026-03-09 19:10:03.810181 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.810185 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.810189 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.810194 | controller | - ' ' 2026-03-09 19:10:03.810198 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810205 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810210 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810214 | controller | - ' ' 2026-03-09 19:10:03.810218 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.810222 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.810227 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.810231 | controller | available"' 2026-03-09 19:10:03.810235 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.810241 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.810246 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.810250 | controller | - ' exit 1' 2026-03-09 19:10:03.810254 | controller | - ' fi' 2026-03-09 19:10:03.810258 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.810263 | controller | - ' ' 2026-03-09 19:10:03.810267 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.810271 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.810275 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810280 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.810284 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810288 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.810293 | controller | -- \' 2026-03-09 19:10:03.810297 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810301 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.810305 | controller | - ' logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.810310 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.810314 | controller | - ' logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.810318 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.810322 | controller | found' 2026-03-09 19:10:03.810335 | controller | - ' logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.810339 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.810344 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.810348 | controller | - ' ' 2026-03-09 19:10:03.810354 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.810359 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.810363 | controller | - ' ' 2026-03-09 19:10:03.810367 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.810372 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.810376 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.810380 | controller | - ' exit 1' 2026-03-09 19:10:03.810384 | controller | - ' fi' 2026-03-09 19:10:03.810389 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.810393 | controller | - ' ' 2026-03-09 19:10:03.810397 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.810401 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.810406 | controller | | base64 -d)' 2026-03-09 19:10:03.810410 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.810414 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.810418 | controller | - ' exit 1' 2026-03-09 19:10:03.810423 | controller | - ' fi' 2026-03-09 19:10:03.810427 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.810431 | controller | - ' ' 2026-03-09 19:10:03.810436 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.810440 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.810446 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.810450 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.810455 | controller | --timeout=300s' 2026-03-09 19:10:03.810460 | controller | - ' ' 2026-03-09 19:10:03.810465 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.810469 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.810473 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.810478 | controller | - ' ' 2026-03-09 19:10:03.810482 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810486 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810490 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810495 | controller | - ' ' 2026-03-09 19:10:03.810499 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.810503 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.810508 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.810512 | controller | available"' 2026-03-09 19:10:03.810516 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.810520 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.810525 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.810529 | controller | - ' exit 1' 2026-03-09 19:10:03.810533 | controller | - ' fi' 2026-03-09 19:10:03.810537 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.810542 | controller | - ' ' 2026-03-09 19:10:03.810546 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.810550 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.810554 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810559 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.810563 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810567 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.810571 | controller | -- \' 2026-03-09 19:10:03.810576 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810580 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.810584 | controller | - ' logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.810588 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.810593 | controller | - ' logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.810597 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.810601 | controller | found' 2026-03-09 19:10:03.810605 | controller | - ' logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.810610 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.810621 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.810628 | controller | - ' ' 2026-03-09 19:10:03.810632 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.810637 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.810641 | controller | - ' ' 2026-03-09 19:10:03.810645 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.810649 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.810654 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.810658 | controller | - ' exit 1' 2026-03-09 19:10:03.810662 | controller | - ' fi' 2026-03-09 19:10:03.810667 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.810671 | controller | - ' ' 2026-03-09 19:10:03.810677 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.810681 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.810686 | controller | | base64 -d)' 2026-03-09 19:10:03.810690 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.810694 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.810698 | controller | - ' exit 1' 2026-03-09 19:10:03.810703 | controller | - ' fi' 2026-03-09 19:10:03.810707 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.810711 | controller | - ' ' 2026-03-09 19:10:03.810716 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.810720 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.810724 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.810728 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.810733 | controller | --timeout=300s' 2026-03-09 19:10:03.810737 | controller | - ' ' 2026-03-09 19:10:03.810741 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.810745 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.810750 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.810754 | controller | - ' ' 2026-03-09 19:10:03.810758 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810763 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810767 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.810771 | controller | - ' ' 2026-03-09 19:10:03.810775 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.810780 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.810784 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.810788 | controller | available"' 2026-03-09 19:10:03.810793 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.810797 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.810801 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.810805 | controller | - ' exit 1' 2026-03-09 19:10:03.810810 | controller | - ' fi' 2026-03-09 19:10:03.810814 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.810818 | controller | - ' ' 2026-03-09 19:10:03.810822 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.810828 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.810833 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810837 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.810841 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810845 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.810850 | controller | -- \' 2026-03-09 19:10:03.810854 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.810858 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.810863 | controller | - ' logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.810867 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.810871 | controller | - ' logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.810875 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.810880 | controller | found' 2026-03-09 19:10:03.810884 | controller | - ' logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.810888 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.810892 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.810897 | controller | - ' ' 2026-03-09 19:10:03.810910 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.810915 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.810919 | controller | - ' ' 2026-03-09 19:10:03.810923 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.810928 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.810932 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.810936 | controller | - ' exit 1' 2026-03-09 19:10:03.810941 | controller | - ' fi' 2026-03-09 19:10:03.810945 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.810949 | controller | - ' ' 2026-03-09 19:10:03.810953 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.810958 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.810962 | controller | | base64 -d)' 2026-03-09 19:10:03.810966 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.810971 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.810975 | controller | - ' exit 1' 2026-03-09 19:10:03.810979 | controller | - ' fi' 2026-03-09 19:10:03.810983 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.810988 | controller | - ' ' 2026-03-09 19:10:03.810992 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.810996 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.811000 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.811005 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.811009 | controller | --timeout=300s' 2026-03-09 19:10:03.811013 | controller | - ' ' 2026-03-09 19:10:03.811017 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.811022 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.811028 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.811032 | controller | - ' ' 2026-03-09 19:10:03.811036 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811040 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811045 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811049 | controller | - ' ' 2026-03-09 19:10:03.811053 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.811058 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.811062 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.811066 | controller | available"' 2026-03-09 19:10:03.811070 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.811075 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.811079 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.811083 | controller | - ' exit 1' 2026-03-09 19:10:03.811088 | controller | - ' fi' 2026-03-09 19:10:03.811092 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.811096 | controller | - ' ' 2026-03-09 19:10:03.811100 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.811105 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.811109 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811113 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.811117 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811122 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.811128 | controller | -- \' 2026-03-09 19:10:03.811132 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811136 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.811142 | controller | - ' logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.811157 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.811163 | controller | - ' logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.811167 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.811171 | controller | found' 2026-03-09 19:10:03.811177 | controller | - ' logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.811182 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.811186 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.811190 | controller | - ' ' 2026-03-09 19:10:03.811202 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.811206 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.811211 | controller | - ' ' 2026-03-09 19:10:03.811215 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.811219 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.811224 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.811230 | controller | - ' exit 1' 2026-03-09 19:10:03.811234 | controller | - ' fi' 2026-03-09 19:10:03.811238 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.811243 | controller | - ' ' 2026-03-09 19:10:03.811247 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.811251 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.811256 | controller | | base64 -d)' 2026-03-09 19:10:03.811260 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.811264 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.811268 | controller | - ' exit 1' 2026-03-09 19:10:03.811273 | controller | - ' fi' 2026-03-09 19:10:03.811277 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.811281 | controller | - ' ' 2026-03-09 19:10:03.811286 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.811290 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.811294 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.811298 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.811303 | controller | --timeout=300s' 2026-03-09 19:10:03.811309 | controller | - ' ' 2026-03-09 19:10:03.811313 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.811317 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.811321 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.811326 | controller | - ' ' 2026-03-09 19:10:03.811330 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811334 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811338 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811343 | controller | - ' ' 2026-03-09 19:10:03.811347 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.811351 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.811355 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.811360 | controller | available"' 2026-03-09 19:10:03.811364 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.811368 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.811373 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.811377 | controller | - ' exit 1' 2026-03-09 19:10:03.811381 | controller | - ' fi' 2026-03-09 19:10:03.811385 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.811390 | controller | - ' ' 2026-03-09 19:10:03.811394 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.811398 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.811402 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811407 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.811411 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.811422 | controller | -- \' 2026-03-09 19:10:03.811427 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811431 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.811435 | controller | - ' logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.811439 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.811444 | controller | - ' logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.811448 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.811452 | controller | found' 2026-03-09 19:10:03.811457 | controller | - ' logger.go:42: 18:59:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.811461 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.811465 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.811469 | controller | - ' ' 2026-03-09 19:10:03.811474 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.811485 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.811490 | controller | - ' ' 2026-03-09 19:10:03.811494 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.811499 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.811503 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.811507 | controller | - ' exit 1' 2026-03-09 19:10:03.811513 | controller | - ' fi' 2026-03-09 19:10:03.811518 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.811522 | controller | - ' ' 2026-03-09 19:10:03.811528 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.811532 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.811537 | controller | | base64 -d)' 2026-03-09 19:10:03.811541 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.811545 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.811549 | controller | - ' exit 1' 2026-03-09 19:10:03.811554 | controller | - ' fi' 2026-03-09 19:10:03.811558 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.811562 | controller | - ' ' 2026-03-09 19:10:03.811567 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.811571 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.811575 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.811580 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.811584 | controller | --timeout=300s' 2026-03-09 19:10:03.811588 | controller | - ' ' 2026-03-09 19:10:03.811592 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.811597 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.811601 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.811605 | controller | - ' ' 2026-03-09 19:10:03.811609 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811614 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811632 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811640 | controller | - ' ' 2026-03-09 19:10:03.811645 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.811649 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.811654 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.811658 | controller | available"' 2026-03-09 19:10:03.811662 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.811666 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.811671 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.811675 | controller | - ' exit 1' 2026-03-09 19:10:03.811679 | controller | - ' fi' 2026-03-09 19:10:03.811683 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.811688 | controller | - ' ' 2026-03-09 19:10:03.811692 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.811696 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.811700 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811704 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.811709 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811713 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.811717 | controller | -- \' 2026-03-09 19:10:03.811722 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811726 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.811730 | controller | - ' logger.go:42: 18:59:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.811734 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.811739 | controller | - ' logger.go:42: 18:59:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.811743 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.811747 | controller | found' 2026-03-09 19:10:03.811751 | controller | - ' logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.811755 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.811760 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.811764 | controller | - ' ' 2026-03-09 19:10:03.811768 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.811780 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.811784 | controller | - ' ' 2026-03-09 19:10:03.811788 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.811793 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.811797 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.811801 | controller | - ' exit 1' 2026-03-09 19:10:03.811805 | controller | - ' fi' 2026-03-09 19:10:03.811810 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.811814 | controller | - ' ' 2026-03-09 19:10:03.811818 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.811823 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.811827 | controller | | base64 -d)' 2026-03-09 19:10:03.811831 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.811837 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.811841 | controller | - ' exit 1' 2026-03-09 19:10:03.811846 | controller | - ' fi' 2026-03-09 19:10:03.811850 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.811854 | controller | - ' ' 2026-03-09 19:10:03.811858 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.811863 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.811867 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.811871 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.811875 | controller | --timeout=300s' 2026-03-09 19:10:03.811880 | controller | - ' ' 2026-03-09 19:10:03.811884 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.811888 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.811892 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.811897 | controller | - ' ' 2026-03-09 19:10:03.811901 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811905 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811910 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.811914 | controller | - ' ' 2026-03-09 19:10:03.811918 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.811922 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.811927 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.811931 | controller | available"' 2026-03-09 19:10:03.811935 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.811941 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.811946 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.811950 | controller | - ' exit 1' 2026-03-09 19:10:03.811954 | controller | - ' fi' 2026-03-09 19:10:03.811958 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.811963 | controller | - ' ' 2026-03-09 19:10:03.811967 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.811971 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.811975 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811980 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.811984 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.811988 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.811992 | controller | -- \' 2026-03-09 19:10:03.811997 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812001 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.812005 | controller | - ' logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.812009 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.812015 | controller | - ' logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.812020 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.812024 | controller | found' 2026-03-09 19:10:03.812028 | controller | - ' logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.812032 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.812037 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.812041 | controller | - ' ' 2026-03-09 19:10:03.812045 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.812049 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.812054 | controller | - ' ' 2026-03-09 19:10:03.812065 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.812069 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.812073 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.812078 | controller | - ' exit 1' 2026-03-09 19:10:03.812082 | controller | - ' fi' 2026-03-09 19:10:03.812086 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.812092 | controller | - ' ' 2026-03-09 19:10:03.812097 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.812101 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.812105 | controller | | base64 -d)' 2026-03-09 19:10:03.812109 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.812114 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.812118 | controller | - ' exit 1' 2026-03-09 19:10:03.812122 | controller | - ' fi' 2026-03-09 19:10:03.812126 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.812131 | controller | - ' ' 2026-03-09 19:10:03.812135 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.812139 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.812143 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.812161 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.812165 | controller | --timeout=300s' 2026-03-09 19:10:03.812170 | controller | - ' ' 2026-03-09 19:10:03.812174 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.812178 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.812182 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.812187 | controller | - ' ' 2026-03-09 19:10:03.812191 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812195 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812201 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812205 | controller | - ' ' 2026-03-09 19:10:03.812210 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.812214 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.812218 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.812222 | controller | available"' 2026-03-09 19:10:03.812227 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.812233 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.812237 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.812241 | controller | - ' exit 1' 2026-03-09 19:10:03.812246 | controller | - ' fi' 2026-03-09 19:10:03.812250 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.812254 | controller | - ' ' 2026-03-09 19:10:03.812258 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.812263 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.812267 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812271 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.812275 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812280 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.812284 | controller | -- \' 2026-03-09 19:10:03.812288 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812293 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.812297 | controller | - ' logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.812301 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.812305 | controller | - ' logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.812310 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.812314 | controller | found' 2026-03-09 19:10:03.812318 | controller | - ' logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.812322 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.812327 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.812331 | controller | - ' ' 2026-03-09 19:10:03.812335 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.812340 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.812344 | controller | - ' ' 2026-03-09 19:10:03.812355 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.812360 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.812364 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.812369 | controller | - ' exit 1' 2026-03-09 19:10:03.812373 | controller | - ' fi' 2026-03-09 19:10:03.812377 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.812382 | controller | - ' ' 2026-03-09 19:10:03.812386 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.812390 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.812394 | controller | | base64 -d)' 2026-03-09 19:10:03.812399 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.812403 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.812407 | controller | - ' exit 1' 2026-03-09 19:10:03.812411 | controller | - ' fi' 2026-03-09 19:10:03.812416 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.812420 | controller | - ' ' 2026-03-09 19:10:03.812424 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.812429 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.812433 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.812439 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.812443 | controller | --timeout=300s' 2026-03-09 19:10:03.812447 | controller | - ' ' 2026-03-09 19:10:03.812452 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.812456 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.812460 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.812464 | controller | - ' ' 2026-03-09 19:10:03.812478 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812482 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812486 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812491 | controller | - ' ' 2026-03-09 19:10:03.812495 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.812499 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.812503 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.812508 | controller | available"' 2026-03-09 19:10:03.812512 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.812516 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.812520 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.812525 | controller | - ' exit 1' 2026-03-09 19:10:03.812529 | controller | - ' fi' 2026-03-09 19:10:03.812533 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.812538 | controller | - ' ' 2026-03-09 19:10:03.812543 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.812548 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.812552 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812556 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.812560 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812565 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.812569 | controller | -- \' 2026-03-09 19:10:03.812573 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812578 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.812582 | controller | - ' logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.812586 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.812590 | controller | - ' logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.812595 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.812599 | controller | found' 2026-03-09 19:10:03.812603 | controller | - ' logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.812607 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.812612 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.812616 | controller | - ' ' 2026-03-09 19:10:03.812620 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.812626 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.812630 | controller | - ' ' 2026-03-09 19:10:03.812635 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.812646 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.812651 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.812655 | controller | - ' exit 1' 2026-03-09 19:10:03.812660 | controller | - ' fi' 2026-03-09 19:10:03.812664 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.812669 | controller | - ' ' 2026-03-09 19:10:03.812673 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.812677 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.812682 | controller | | base64 -d)' 2026-03-09 19:10:03.812686 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.812690 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.812694 | controller | - ' exit 1' 2026-03-09 19:10:03.812699 | controller | - ' fi' 2026-03-09 19:10:03.812703 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.812707 | controller | - ' ' 2026-03-09 19:10:03.812712 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.812716 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.812722 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.812726 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.812730 | controller | --timeout=300s' 2026-03-09 19:10:03.812735 | controller | - ' ' 2026-03-09 19:10:03.812739 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.812743 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.812747 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.812752 | controller | - ' ' 2026-03-09 19:10:03.812756 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812760 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812764 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.812769 | controller | - ' ' 2026-03-09 19:10:03.812773 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.812777 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.812781 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.812786 | controller | available"' 2026-03-09 19:10:03.812790 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.812794 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.812798 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.812803 | controller | - ' exit 1' 2026-03-09 19:10:03.812807 | controller | - ' fi' 2026-03-09 19:10:03.812811 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.812815 | controller | - ' ' 2026-03-09 19:10:03.812820 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.812824 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.812831 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812836 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.812840 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812844 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.812849 | controller | -- \' 2026-03-09 19:10:03.812853 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.812857 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.812861 | controller | - ' logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.812866 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.812870 | controller | - ' logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.812874 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.812879 | controller | found' 2026-03-09 19:10:03.812883 | controller | - ' logger.go:42: 19:00:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.812887 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.812891 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.812896 | controller | - ' ' 2026-03-09 19:10:03.812900 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.812904 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.812909 | controller | - ' ' 2026-03-09 19:10:03.812913 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.812917 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.812929 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.812933 | controller | - ' exit 1' 2026-03-09 19:10:03.812938 | controller | - ' fi' 2026-03-09 19:10:03.812942 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.812946 | controller | - ' ' 2026-03-09 19:10:03.812950 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.812955 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.812959 | controller | | base64 -d)' 2026-03-09 19:10:03.812963 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.812968 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.812972 | controller | - ' exit 1' 2026-03-09 19:10:03.812976 | controller | - ' fi' 2026-03-09 19:10:03.812980 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.812985 | controller | - ' ' 2026-03-09 19:10:03.812989 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.812993 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.812997 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.813002 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.813006 | controller | --timeout=300s' 2026-03-09 19:10:03.813010 | controller | - ' ' 2026-03-09 19:10:03.813015 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.813019 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.813025 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.813029 | controller | - ' ' 2026-03-09 19:10:03.813033 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813038 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813042 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813046 | controller | - ' ' 2026-03-09 19:10:03.813050 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.813055 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.813059 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.813063 | controller | available"' 2026-03-09 19:10:03.813068 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.813072 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.813076 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.813080 | controller | - ' exit 1' 2026-03-09 19:10:03.813085 | controller | - ' fi' 2026-03-09 19:10:03.813089 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.813093 | controller | - ' ' 2026-03-09 19:10:03.813097 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.813102 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.813106 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813110 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.813114 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813119 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.813123 | controller | -- \' 2026-03-09 19:10:03.813127 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813132 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.813138 | controller | - ' logger.go:42: 19:00:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.813142 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.813156 | controller | - ' logger.go:42: 19:00:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.813163 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.813167 | controller | found' 2026-03-09 19:10:03.813171 | controller | - ' logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.813176 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.813180 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.813184 | controller | - ' ' 2026-03-09 19:10:03.813188 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.813193 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.813197 | controller | - ' ' 2026-03-09 19:10:03.813201 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.813205 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.813210 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.813221 | controller | - ' exit 1' 2026-03-09 19:10:03.813228 | controller | - ' fi' 2026-03-09 19:10:03.813232 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.813237 | controller | - ' ' 2026-03-09 19:10:03.813241 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.813245 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.813250 | controller | | base64 -d)' 2026-03-09 19:10:03.813254 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.813258 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.813264 | controller | - ' exit 1' 2026-03-09 19:10:03.813269 | controller | - ' fi' 2026-03-09 19:10:03.813273 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.813277 | controller | - ' ' 2026-03-09 19:10:03.813282 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.813286 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.813290 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.813294 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.813301 | controller | --timeout=300s' 2026-03-09 19:10:03.813305 | controller | - ' ' 2026-03-09 19:10:03.813309 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.813313 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.813318 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.813322 | controller | - ' ' 2026-03-09 19:10:03.813326 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813331 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813335 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813339 | controller | - ' ' 2026-03-09 19:10:03.813343 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.813348 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.813352 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.813356 | controller | available"' 2026-03-09 19:10:03.813361 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.813365 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.813369 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.813373 | controller | - ' exit 1' 2026-03-09 19:10:03.813378 | controller | - ' fi' 2026-03-09 19:10:03.813382 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.813386 | controller | - ' ' 2026-03-09 19:10:03.813390 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.813395 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.813399 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813403 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.813407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813412 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.813418 | controller | -- \' 2026-03-09 19:10:03.813422 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813426 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.813430 | controller | - ' logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.813435 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.813439 | controller | - ' logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.813443 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.813448 | controller | found' 2026-03-09 19:10:03.813452 | controller | - ' logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.813456 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.813461 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.813467 | controller | - ' ' 2026-03-09 19:10:03.813471 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.813475 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.813480 | controller | - ' ' 2026-03-09 19:10:03.813484 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.813489 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.813493 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.813497 | controller | - ' exit 1' 2026-03-09 19:10:03.813502 | controller | - ' fi' 2026-03-09 19:10:03.813513 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.813517 | controller | - ' ' 2026-03-09 19:10:03.813522 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.813526 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.813532 | controller | | base64 -d)' 2026-03-09 19:10:03.813536 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.813541 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.813545 | controller | - ' exit 1' 2026-03-09 19:10:03.813549 | controller | - ' fi' 2026-03-09 19:10:03.813554 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.813558 | controller | - ' ' 2026-03-09 19:10:03.813562 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.813567 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.813571 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.813575 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.813580 | controller | --timeout=300s' 2026-03-09 19:10:03.813584 | controller | - ' ' 2026-03-09 19:10:03.813588 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.813593 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.813597 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.813601 | controller | - ' ' 2026-03-09 19:10:03.813607 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813612 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813616 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813622 | controller | - ' ' 2026-03-09 19:10:03.813626 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.813631 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.813635 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.813639 | controller | available"' 2026-03-09 19:10:03.813644 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.813648 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.813653 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.813657 | controller | - ' exit 1' 2026-03-09 19:10:03.813661 | controller | - ' fi' 2026-03-09 19:10:03.813665 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.813670 | controller | - ' ' 2026-03-09 19:10:03.813674 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.813678 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.813683 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813687 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.813691 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813696 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.813700 | controller | -- \' 2026-03-09 19:10:03.813704 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813709 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.813713 | controller | - ' logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.813717 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.813722 | controller | - ' logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.813726 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.813730 | controller | found' 2026-03-09 19:10:03.813735 | controller | - ' logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.813739 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.813743 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.813747 | controller | - ' ' 2026-03-09 19:10:03.813752 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.813756 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.813760 | controller | - ' ' 2026-03-09 19:10:03.813764 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.813768 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.813773 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.813777 | controller | - ' exit 1' 2026-03-09 19:10:03.813781 | controller | - ' fi' 2026-03-09 19:10:03.813786 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.813790 | controller | - ' ' 2026-03-09 19:10:03.813801 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.813805 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.813810 | controller | | base64 -d)' 2026-03-09 19:10:03.813814 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.813820 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.813826 | controller | - ' exit 1' 2026-03-09 19:10:03.813830 | controller | - ' fi' 2026-03-09 19:10:03.813834 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.813839 | controller | - ' ' 2026-03-09 19:10:03.813843 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.813847 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.813851 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.813856 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.813860 | controller | --timeout=300s' 2026-03-09 19:10:03.813864 | controller | - ' ' 2026-03-09 19:10:03.813869 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.813873 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.813877 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.813881 | controller | - ' ' 2026-03-09 19:10:03.813886 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813890 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813894 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.813898 | controller | - ' ' 2026-03-09 19:10:03.813904 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.813909 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.813913 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.813917 | controller | available"' 2026-03-09 19:10:03.813921 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.813926 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.813930 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.813934 | controller | - ' exit 1' 2026-03-09 19:10:03.813938 | controller | - ' fi' 2026-03-09 19:10:03.813943 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.813947 | controller | - ' ' 2026-03-09 19:10:03.813951 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.813956 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.813960 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813964 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.813968 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813973 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.813977 | controller | -- \' 2026-03-09 19:10:03.813981 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.813985 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.813990 | controller | - ' logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.813994 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.813998 | controller | - ' logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.814004 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.814008 | controller | found' 2026-03-09 19:10:03.814013 | controller | - ' logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.814017 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.814021 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.814026 | controller | - ' ' 2026-03-09 19:10:03.814030 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.814034 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.814038 | controller | - ' ' 2026-03-09 19:10:03.814043 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.814047 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.814051 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.814055 | controller | - ' exit 1' 2026-03-09 19:10:03.814060 | controller | - ' fi' 2026-03-09 19:10:03.814064 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.814068 | controller | - ' ' 2026-03-09 19:10:03.814079 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.814083 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.814088 | controller | | base64 -d)' 2026-03-09 19:10:03.814092 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.814096 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.814101 | controller | - ' exit 1' 2026-03-09 19:10:03.814105 | controller | - ' fi' 2026-03-09 19:10:03.814109 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.814114 | controller | - ' ' 2026-03-09 19:10:03.814118 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.814122 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.814126 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.814131 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.814135 | controller | --timeout=300s' 2026-03-09 19:10:03.814139 | controller | - ' ' 2026-03-09 19:10:03.814144 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.814161 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.814165 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.814169 | controller | - ' ' 2026-03-09 19:10:03.814174 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814178 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814182 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814186 | controller | - ' ' 2026-03-09 19:10:03.814191 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.814198 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.814202 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.814207 | controller | available"' 2026-03-09 19:10:03.814211 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.814215 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.814221 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.814226 | controller | - ' exit 1' 2026-03-09 19:10:03.814230 | controller | - ' fi' 2026-03-09 19:10:03.814234 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.814238 | controller | - ' ' 2026-03-09 19:10:03.814243 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.814247 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.814251 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814256 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.814260 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814264 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.814268 | controller | -- \' 2026-03-09 19:10:03.814273 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814277 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.814281 | controller | - ' logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.814285 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.814290 | controller | - ' logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.814294 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.814298 | controller | found' 2026-03-09 19:10:03.814302 | controller | - ' logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.814307 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.814311 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.814315 | controller | - ' ' 2026-03-09 19:10:03.814319 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.814324 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.814328 | controller | - ' ' 2026-03-09 19:10:03.814332 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.814336 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.814341 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.814345 | controller | - ' exit 1' 2026-03-09 19:10:03.814349 | controller | - ' fi' 2026-03-09 19:10:03.814354 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.814358 | controller | - ' ' 2026-03-09 19:10:03.814397 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.814410 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.814415 | controller | | base64 -d)' 2026-03-09 19:10:03.814419 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.814423 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.814428 | controller | - ' exit 1' 2026-03-09 19:10:03.814432 | controller | - ' fi' 2026-03-09 19:10:03.814436 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.814440 | controller | - ' ' 2026-03-09 19:10:03.814445 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.814449 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.814453 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.814460 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.814464 | controller | --timeout=300s' 2026-03-09 19:10:03.814468 | controller | - ' ' 2026-03-09 19:10:03.814473 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.814477 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.814481 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.814485 | controller | - ' ' 2026-03-09 19:10:03.814490 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814494 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814498 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814502 | controller | - ' ' 2026-03-09 19:10:03.814507 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.814511 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.814517 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.814521 | controller | available"' 2026-03-09 19:10:03.814526 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.814531 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.814536 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.814540 | controller | - ' exit 1' 2026-03-09 19:10:03.814544 | controller | - ' fi' 2026-03-09 19:10:03.814548 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.814553 | controller | - ' ' 2026-03-09 19:10:03.814557 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.814561 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.814565 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814570 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.814574 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814578 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.814583 | controller | -- \' 2026-03-09 19:10:03.814587 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814591 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.814595 | controller | - ' logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.814600 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.814604 | controller | - ' logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.814608 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.814612 | controller | found' 2026-03-09 19:10:03.814617 | controller | - ' logger.go:42: 19:00:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.814621 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.814625 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.814630 | controller | - ' ' 2026-03-09 19:10:03.814634 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.814640 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.814644 | controller | - ' ' 2026-03-09 19:10:03.814648 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.814653 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.814657 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.814661 | controller | - ' exit 1' 2026-03-09 19:10:03.814665 | controller | - ' fi' 2026-03-09 19:10:03.814670 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.814674 | controller | - ' ' 2026-03-09 19:10:03.814678 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.814689 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.814694 | controller | | base64 -d)' 2026-03-09 19:10:03.814698 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.814702 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.814707 | controller | - ' exit 1' 2026-03-09 19:10:03.814712 | controller | - ' fi' 2026-03-09 19:10:03.814717 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.814721 | controller | - ' ' 2026-03-09 19:10:03.814725 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.814730 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.814734 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.814738 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.814742 | controller | --timeout=300s' 2026-03-09 19:10:03.814747 | controller | - ' ' 2026-03-09 19:10:03.814751 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.814755 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.814759 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.814764 | controller | - ' ' 2026-03-09 19:10:03.814768 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814772 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814776 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.814781 | controller | - ' ' 2026-03-09 19:10:03.814785 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.814789 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.814793 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.814798 | controller | available"' 2026-03-09 19:10:03.814802 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.814806 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.814810 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.814815 | controller | - ' exit 1' 2026-03-09 19:10:03.814819 | controller | - ' fi' 2026-03-09 19:10:03.814823 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.814828 | controller | - ' ' 2026-03-09 19:10:03.814832 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.814836 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.814842 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814846 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.814850 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814855 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.814859 | controller | -- \' 2026-03-09 19:10:03.814863 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.814868 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.814895 | controller | - ' logger.go:42: 19:00:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.814901 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.814905 | controller | - ' logger.go:42: 19:00:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.814909 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.814913 | controller | found' 2026-03-09 19:10:03.814918 | controller | - ' logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.814922 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.814926 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.814931 | controller | - ' ' 2026-03-09 19:10:03.814935 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.814939 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.814943 | controller | - ' ' 2026-03-09 19:10:03.814948 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.814952 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.814956 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.814960 | controller | - ' exit 1' 2026-03-09 19:10:03.814965 | controller | - ' fi' 2026-03-09 19:10:03.814969 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.814973 | controller | - ' ' 2026-03-09 19:10:03.814977 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.814982 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.814986 | controller | | base64 -d)' 2026-03-09 19:10:03.815109 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.815113 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.815117 | controller | - ' exit 1' 2026-03-09 19:10:03.815122 | controller | - ' fi' 2026-03-09 19:10:03.815126 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.815130 | controller | - ' ' 2026-03-09 19:10:03.815135 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.815139 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.815143 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.815159 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.815164 | controller | --timeout=300s' 2026-03-09 19:10:03.815168 | controller | - ' ' 2026-03-09 19:10:03.815172 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.815177 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.815181 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.815187 | controller | - ' ' 2026-03-09 19:10:03.815192 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815196 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815200 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815205 | controller | - ' ' 2026-03-09 19:10:03.815209 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.815213 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.815218 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.815222 | controller | available"' 2026-03-09 19:10:03.815226 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.815234 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.815238 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.815243 | controller | - ' exit 1' 2026-03-09 19:10:03.815247 | controller | - ' fi' 2026-03-09 19:10:03.815251 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.815255 | controller | - ' ' 2026-03-09 19:10:03.815260 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.815264 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.815268 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815272 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.815277 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815281 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.815285 | controller | -- \' 2026-03-09 19:10:03.815289 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815294 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.815300 | controller | - ' logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.815304 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.815308 | controller | - ' logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.815312 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.815317 | controller | found' 2026-03-09 19:10:03.815321 | controller | - ' logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.815325 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.815329 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.815334 | controller | - ' ' 2026-03-09 19:10:03.815338 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.815342 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.815346 | controller | - ' ' 2026-03-09 19:10:03.815351 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.815355 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.815359 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.815364 | controller | - ' exit 1' 2026-03-09 19:10:03.815368 | controller | - ' fi' 2026-03-09 19:10:03.815372 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.815378 | controller | - ' ' 2026-03-09 19:10:03.815383 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.815387 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.815391 | controller | | base64 -d)' 2026-03-09 19:10:03.815395 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.815408 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.815412 | controller | - ' exit 1' 2026-03-09 19:10:03.815417 | controller | - ' fi' 2026-03-09 19:10:03.815421 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.815425 | controller | - ' ' 2026-03-09 19:10:03.815430 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.815434 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.815438 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.815443 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.815447 | controller | --timeout=300s' 2026-03-09 19:10:03.815453 | controller | - ' ' 2026-03-09 19:10:03.815457 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.815462 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.815466 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.815470 | controller | - ' ' 2026-03-09 19:10:03.815474 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815479 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815483 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815487 | controller | - ' ' 2026-03-09 19:10:03.815492 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.815496 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.815500 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.815504 | controller | available"' 2026-03-09 19:10:03.815509 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.815513 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.815517 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.815521 | controller | - ' exit 1' 2026-03-09 19:10:03.815527 | controller | - ' fi' 2026-03-09 19:10:03.815532 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.815536 | controller | - ' ' 2026-03-09 19:10:03.815540 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.815544 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.815549 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815553 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.815557 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815562 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.815566 | controller | -- \' 2026-03-09 19:10:03.815572 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815576 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.815580 | controller | - ' logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.815585 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.815589 | controller | - ' logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.815593 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.815598 | controller | found' 2026-03-09 19:10:03.815602 | controller | - ' logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.815606 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.815610 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.815629 | controller | - ' ' 2026-03-09 19:10:03.815635 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.815641 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.815645 | controller | - ' ' 2026-03-09 19:10:03.815649 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.815653 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.815658 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.815662 | controller | - ' exit 1' 2026-03-09 19:10:03.815666 | controller | - ' fi' 2026-03-09 19:10:03.815671 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.815675 | controller | - ' ' 2026-03-09 19:10:03.815679 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.815683 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.815688 | controller | | base64 -d)' 2026-03-09 19:10:03.815692 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.815703 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.815708 | controller | - ' exit 1' 2026-03-09 19:10:03.815712 | controller | - ' fi' 2026-03-09 19:10:03.815716 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.815721 | controller | - ' ' 2026-03-09 19:10:03.815725 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.815729 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.815734 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.815738 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.815742 | controller | --timeout=300s' 2026-03-09 19:10:03.815746 | controller | - ' ' 2026-03-09 19:10:03.815751 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.815755 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.815759 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.815763 | controller | - ' ' 2026-03-09 19:10:03.815768 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815772 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815776 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.815781 | controller | - ' ' 2026-03-09 19:10:03.815785 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.815792 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.815797 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.815801 | controller | available"' 2026-03-09 19:10:03.815805 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.815810 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.815814 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.815818 | controller | - ' exit 1' 2026-03-09 19:10:03.815822 | controller | - ' fi' 2026-03-09 19:10:03.815827 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.815831 | controller | - ' ' 2026-03-09 19:10:03.815837 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.815841 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.815846 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815850 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.815854 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815859 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.815863 | controller | -- \' 2026-03-09 19:10:03.815867 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.815871 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.815876 | controller | - ' logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.815880 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.815884 | controller | - ' logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.815888 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.815893 | controller | found' 2026-03-09 19:10:03.815897 | controller | - ' logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.815901 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.815905 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.815910 | controller | - ' ' 2026-03-09 19:10:03.815914 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.815918 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.815922 | controller | - ' ' 2026-03-09 19:10:03.815928 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.815933 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.815937 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.815941 | controller | - ' exit 1' 2026-03-09 19:10:03.815945 | controller | - ' fi' 2026-03-09 19:10:03.815950 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.815954 | controller | - ' ' 2026-03-09 19:10:03.815960 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.815964 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.815969 | controller | | base64 -d)' 2026-03-09 19:10:03.815973 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.815977 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.816130 | controller | - ' exit 1' 2026-03-09 19:10:03.816141 | controller | - ' fi' 2026-03-09 19:10:03.816158 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.816164 | controller | - ' ' 2026-03-09 19:10:03.816169 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.816173 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.816177 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.816181 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.816186 | controller | --timeout=300s' 2026-03-09 19:10:03.816190 | controller | - ' ' 2026-03-09 19:10:03.816194 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.816199 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.816203 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.816209 | controller | - ' ' 2026-03-09 19:10:03.816214 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816218 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816222 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816226 | controller | - ' ' 2026-03-09 19:10:03.816231 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.816235 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.816239 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.816244 | controller | available"' 2026-03-09 19:10:03.816248 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.816252 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.816256 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.816261 | controller | - ' exit 1' 2026-03-09 19:10:03.816265 | controller | - ' fi' 2026-03-09 19:10:03.816269 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.816273 | controller | - ' ' 2026-03-09 19:10:03.816278 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.816282 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.816286 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816290 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.816295 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816299 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.816303 | controller | -- \' 2026-03-09 19:10:03.816307 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816312 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.816316 | controller | - ' logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.816320 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.816325 | controller | - ' logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.816329 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.816335 | controller | found' 2026-03-09 19:10:03.816339 | controller | - ' logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.816343 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.816348 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.816352 | controller | - ' ' 2026-03-09 19:10:03.816356 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.816360 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.816365 | controller | - ' ' 2026-03-09 19:10:03.816369 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.816373 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.816378 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.816382 | controller | - ' exit 1' 2026-03-09 19:10:03.816386 | controller | - ' fi' 2026-03-09 19:10:03.816390 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.816395 | controller | - ' ' 2026-03-09 19:10:03.816399 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.816403 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.816407 | controller | | base64 -d)' 2026-03-09 19:10:03.816412 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.816416 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.816420 | controller | - ' exit 1' 2026-03-09 19:10:03.816424 | controller | - ' fi' 2026-03-09 19:10:03.816436 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.816441 | controller | - ' ' 2026-03-09 19:10:03.816445 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.816449 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.816453 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.816458 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.816462 | controller | --timeout=300s' 2026-03-09 19:10:03.816466 | controller | - ' ' 2026-03-09 19:10:03.816471 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.816475 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.816479 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.816483 | controller | - ' ' 2026-03-09 19:10:03.816488 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816492 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816496 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816501 | controller | - ' ' 2026-03-09 19:10:03.816505 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.816509 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.816513 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.816518 | controller | available"' 2026-03-09 19:10:03.816522 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.816526 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.816530 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.816536 | controller | - ' exit 1' 2026-03-09 19:10:03.816541 | controller | - ' fi' 2026-03-09 19:10:03.816545 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.816549 | controller | - ' ' 2026-03-09 19:10:03.816553 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.816558 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.816562 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816568 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.816574 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816578 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.816582 | controller | -- \' 2026-03-09 19:10:03.816586 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816591 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.816595 | controller | - ' logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.816599 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.816603 | controller | - ' logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.816608 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.816612 | controller | found' 2026-03-09 19:10:03.816616 | controller | - ' logger.go:42: 19:00:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.816620 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.816625 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.816629 | controller | - ' ' 2026-03-09 19:10:03.816633 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.816637 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.816642 | controller | - ' ' 2026-03-09 19:10:03.816646 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.816650 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.816654 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.816659 | controller | - ' exit 1' 2026-03-09 19:10:03.816663 | controller | - ' fi' 2026-03-09 19:10:03.816667 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.816671 | controller | - ' ' 2026-03-09 19:10:03.816676 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.816680 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.816684 | controller | | base64 -d)' 2026-03-09 19:10:03.816688 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.816693 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.816697 | controller | - ' exit 1' 2026-03-09 19:10:03.816701 | controller | - ' fi' 2026-03-09 19:10:03.816705 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.816710 | controller | - ' ' 2026-03-09 19:10:03.816721 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.816725 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.816729 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.816734 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.816740 | controller | --timeout=300s' 2026-03-09 19:10:03.816744 | controller | - ' ' 2026-03-09 19:10:03.816748 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.816753 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.816757 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.816761 | controller | - ' ' 2026-03-09 19:10:03.816767 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816772 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816776 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.816780 | controller | - ' ' 2026-03-09 19:10:03.816784 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.816789 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.816793 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.816797 | controller | available"' 2026-03-09 19:10:03.816802 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.816806 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.816810 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.816814 | controller | - ' exit 1' 2026-03-09 19:10:03.816819 | controller | - ' fi' 2026-03-09 19:10:03.816823 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.816827 | controller | - ' ' 2026-03-09 19:10:03.816831 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.816836 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.816840 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816844 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.816848 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816853 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.816857 | controller | -- \' 2026-03-09 19:10:03.816861 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.816865 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.816870 | controller | - ' logger.go:42: 19:00:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.816874 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.816878 | controller | - ' logger.go:42: 19:00:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.816883 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.816887 | controller | found' 2026-03-09 19:10:03.816891 | controller | - ' logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.816895 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.816900 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.816904 | controller | - ' ' 2026-03-09 19:10:03.816908 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.816912 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.816918 | controller | - ' ' 2026-03-09 19:10:03.816923 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.816927 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.816931 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.816935 | controller | - ' exit 1' 2026-03-09 19:10:03.816940 | controller | - ' fi' 2026-03-09 19:10:03.816944 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.816948 | controller | - ' ' 2026-03-09 19:10:03.816953 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.816957 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.816961 | controller | | base64 -d)' 2026-03-09 19:10:03.816965 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.816970 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.816974 | controller | - ' exit 1' 2026-03-09 19:10:03.816978 | controller | - ' fi' 2026-03-09 19:10:03.816982 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.816987 | controller | - ' ' 2026-03-09 19:10:03.816991 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.817002 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.817006 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.817010 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.817015 | controller | --timeout=300s' 2026-03-09 19:10:03.817021 | controller | - ' ' 2026-03-09 19:10:03.817027 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.817031 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.817035 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.817040 | controller | - ' ' 2026-03-09 19:10:03.817044 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817049 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817055 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817060 | controller | - ' ' 2026-03-09 19:10:03.817064 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.817068 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.817072 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.817077 | controller | available"' 2026-03-09 19:10:03.817083 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.817087 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.817093 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.817098 | controller | - ' exit 1' 2026-03-09 19:10:03.817103 | controller | - ' fi' 2026-03-09 19:10:03.817107 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.817111 | controller | - ' ' 2026-03-09 19:10:03.817115 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.817120 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.817124 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817130 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.817134 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817139 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.817143 | controller | -- \' 2026-03-09 19:10:03.817160 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817168 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.817172 | controller | - ' logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.817176 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.817180 | controller | - ' logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.817185 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.817189 | controller | found' 2026-03-09 19:10:03.817193 | controller | - ' logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.817197 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.817202 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.817206 | controller | - ' ' 2026-03-09 19:10:03.817210 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.817214 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.817219 | controller | - ' ' 2026-03-09 19:10:03.817223 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.817227 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.817232 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.817236 | controller | - ' exit 1' 2026-03-09 19:10:03.817240 | controller | - ' fi' 2026-03-09 19:10:03.817244 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.817249 | controller | - ' ' 2026-03-09 19:10:03.817253 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.817257 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.817262 | controller | | base64 -d)' 2026-03-09 19:10:03.817266 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.817270 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.817274 | controller | - ' exit 1' 2026-03-09 19:10:03.817279 | controller | - ' fi' 2026-03-09 19:10:03.817283 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.817287 | controller | - ' ' 2026-03-09 19:10:03.817292 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.817303 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.817308 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.817312 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.817317 | controller | --timeout=300s' 2026-03-09 19:10:03.817321 | controller | - ' ' 2026-03-09 19:10:03.817325 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.817330 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.817334 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.817340 | controller | - ' ' 2026-03-09 19:10:03.817344 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817349 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817353 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817357 | controller | - ' ' 2026-03-09 19:10:03.817361 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.817366 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.817370 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.817374 | controller | available"' 2026-03-09 19:10:03.817379 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.817385 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.817390 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.817394 | controller | - ' exit 1' 2026-03-09 19:10:03.817398 | controller | - ' fi' 2026-03-09 19:10:03.817403 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.817407 | controller | - ' ' 2026-03-09 19:10:03.817411 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.817415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.817420 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817424 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.817428 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817433 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.817437 | controller | -- \' 2026-03-09 19:10:03.817441 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817446 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.817450 | controller | - ' logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.817454 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.817460 | controller | - ' logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.817464 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.817469 | controller | found' 2026-03-09 19:10:03.817473 | controller | - ' logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.817477 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.817482 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.817486 | controller | - ' ' 2026-03-09 19:10:03.817490 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.817495 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.817499 | controller | - ' ' 2026-03-09 19:10:03.817503 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.817507 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.817512 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.817516 | controller | - ' exit 1' 2026-03-09 19:10:03.817520 | controller | - ' fi' 2026-03-09 19:10:03.817524 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.817529 | controller | - ' ' 2026-03-09 19:10:03.817535 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.817539 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.817543 | controller | | base64 -d)' 2026-03-09 19:10:03.817547 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.817552 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.817556 | controller | - ' exit 1' 2026-03-09 19:10:03.817560 | controller | - ' fi' 2026-03-09 19:10:03.817564 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.817569 | controller | - ' ' 2026-03-09 19:10:03.817573 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.817577 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.817589 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.817594 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.817598 | controller | --timeout=300s' 2026-03-09 19:10:03.817602 | controller | - ' ' 2026-03-09 19:10:03.817607 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.817611 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.817615 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.817619 | controller | - ' ' 2026-03-09 19:10:03.817624 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817630 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817634 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817638 | controller | - ' ' 2026-03-09 19:10:03.817643 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.817647 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.817651 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.817655 | controller | available"' 2026-03-09 19:10:03.817660 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.817664 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.817668 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.817672 | controller | - ' exit 1' 2026-03-09 19:10:03.817677 | controller | - ' fi' 2026-03-09 19:10:03.817681 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.817685 | controller | - ' ' 2026-03-09 19:10:03.817689 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.817694 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.817698 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817702 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.817706 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817711 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.817715 | controller | -- \' 2026-03-09 19:10:03.817719 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817725 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.817730 | controller | - ' logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.817734 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.817740 | controller | - ' logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.817744 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.817748 | controller | found' 2026-03-09 19:10:03.817752 | controller | - ' logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.817757 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.817761 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.817765 | controller | - ' ' 2026-03-09 19:10:03.817769 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.817774 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.817778 | controller | - ' ' 2026-03-09 19:10:03.817782 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.817786 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.817791 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.817796 | controller | - ' exit 1' 2026-03-09 19:10:03.817800 | controller | - ' fi' 2026-03-09 19:10:03.817804 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.817808 | controller | - ' ' 2026-03-09 19:10:03.817813 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.817817 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.817822 | controller | | base64 -d)' 2026-03-09 19:10:03.817827 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.817833 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.817837 | controller | - ' exit 1' 2026-03-09 19:10:03.817841 | controller | - ' fi' 2026-03-09 19:10:03.817846 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.817850 | controller | - ' ' 2026-03-09 19:10:03.817854 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.817858 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.817870 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.817874 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.817878 | controller | --timeout=300s' 2026-03-09 19:10:03.817883 | controller | - ' ' 2026-03-09 19:10:03.817887 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.817891 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.817896 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.817900 | controller | - ' ' 2026-03-09 19:10:03.817904 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817908 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817913 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.817917 | controller | - ' ' 2026-03-09 19:10:03.817921 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.817927 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.817932 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.817936 | controller | available"' 2026-03-09 19:10:03.817940 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.817945 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.817949 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.817953 | controller | - ' exit 1' 2026-03-09 19:10:03.817957 | controller | - ' fi' 2026-03-09 19:10:03.817962 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.817966 | controller | - ' ' 2026-03-09 19:10:03.817970 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.817975 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.817979 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817983 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.817987 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.817992 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.817996 | controller | -- \' 2026-03-09 19:10:03.818000 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818005 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.818009 | controller | - ' logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.818013 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.818017 | controller | - ' logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.818022 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.818026 | controller | found' 2026-03-09 19:10:03.818030 | controller | - ' logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.818035 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.818039 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.818043 | controller | - ' ' 2026-03-09 19:10:03.818047 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.818052 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.818056 | controller | - ' ' 2026-03-09 19:10:03.818060 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.818065 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.818069 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.818073 | controller | - ' exit 1' 2026-03-09 19:10:03.818078 | controller | - ' fi' 2026-03-09 19:10:03.818082 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.818086 | controller | - ' ' 2026-03-09 19:10:03.818090 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.818095 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.818099 | controller | | base64 -d)' 2026-03-09 19:10:03.818103 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.818114 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.818118 | controller | - ' exit 1' 2026-03-09 19:10:03.818123 | controller | - ' fi' 2026-03-09 19:10:03.818128 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.818133 | controller | - ' ' 2026-03-09 19:10:03.818138 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.818186 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.818196 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.818208 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.818213 | controller | --timeout=300s' 2026-03-09 19:10:03.818217 | controller | - ' ' 2026-03-09 19:10:03.818222 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.818226 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.818230 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.818235 | controller | - ' ' 2026-03-09 19:10:03.818263 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818268 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818273 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818277 | controller | - ' ' 2026-03-09 19:10:03.818281 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.818286 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.818290 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.818294 | controller | available"' 2026-03-09 19:10:03.818298 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.818303 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.818307 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.818311 | controller | - ' exit 1' 2026-03-09 19:10:03.818316 | controller | - ' fi' 2026-03-09 19:10:03.818320 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.818324 | controller | - ' ' 2026-03-09 19:10:03.818328 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.818333 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.818337 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818341 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.818345 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818350 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.818354 | controller | -- \' 2026-03-09 19:10:03.818358 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818363 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.818367 | controller | - ' logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.818371 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.818375 | controller | - ' logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.818383 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.818389 | controller | found' 2026-03-09 19:10:03.818393 | controller | - ' logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.818398 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.818402 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.818406 | controller | - ' ' 2026-03-09 19:10:03.818410 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.818415 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.818419 | controller | - ' ' 2026-03-09 19:10:03.818423 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.818427 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.818432 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.818436 | controller | - ' exit 1' 2026-03-09 19:10:03.818440 | controller | - ' fi' 2026-03-09 19:10:03.818445 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.818449 | controller | - ' ' 2026-03-09 19:10:03.818453 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.818457 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.818462 | controller | | base64 -d)' 2026-03-09 19:10:03.818466 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.818470 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.818474 | controller | - ' exit 1' 2026-03-09 19:10:03.818479 | controller | - ' fi' 2026-03-09 19:10:03.818483 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.818487 | controller | - ' ' 2026-03-09 19:10:03.818492 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.818496 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.818500 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.818504 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.818516 | controller | --timeout=300s' 2026-03-09 19:10:03.818522 | controller | - ' ' 2026-03-09 19:10:03.818527 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.818533 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.818537 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.818541 | controller | - ' ' 2026-03-09 19:10:03.818546 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818550 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818554 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818558 | controller | - ' ' 2026-03-09 19:10:03.818563 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.818567 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.818571 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.818576 | controller | available"' 2026-03-09 19:10:03.818580 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.818584 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.818588 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.818602 | controller | - ' exit 1' 2026-03-09 19:10:03.818607 | controller | - ' fi' 2026-03-09 19:10:03.818611 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.818615 | controller | - ' ' 2026-03-09 19:10:03.818623 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.818628 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.818632 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818636 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.818641 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818645 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.818651 | controller | -- \' 2026-03-09 19:10:03.818655 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818659 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.818664 | controller | - ' logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.818668 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.818672 | controller | - ' logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.818677 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.818681 | controller | found' 2026-03-09 19:10:03.818685 | controller | - ' logger.go:42: 19:00:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.818689 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.818694 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.818698 | controller | - ' ' 2026-03-09 19:10:03.818702 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.818707 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.818711 | controller | - ' ' 2026-03-09 19:10:03.818715 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.818719 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.818724 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.818728 | controller | - ' exit 1' 2026-03-09 19:10:03.818732 | controller | - ' fi' 2026-03-09 19:10:03.818737 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.818741 | controller | - ' ' 2026-03-09 19:10:03.818747 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.818751 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.818755 | controller | | base64 -d)' 2026-03-09 19:10:03.818760 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.818764 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.818768 | controller | - ' exit 1' 2026-03-09 19:10:03.818772 | controller | - ' fi' 2026-03-09 19:10:03.818777 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.818781 | controller | - ' ' 2026-03-09 19:10:03.818785 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.818789 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.818794 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.818798 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.818804 | controller | --timeout=300s' 2026-03-09 19:10:03.818808 | controller | - ' ' 2026-03-09 19:10:03.818820 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.818825 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.818829 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.818833 | controller | - ' ' 2026-03-09 19:10:03.818838 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818843 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818849 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.818855 | controller | - ' ' 2026-03-09 19:10:03.818861 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.818867 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.818872 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.818877 | controller | available"' 2026-03-09 19:10:03.818881 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.818885 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.818890 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.818894 | controller | - ' exit 1' 2026-03-09 19:10:03.818898 | controller | - ' fi' 2026-03-09 19:10:03.818902 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.818907 | controller | - ' ' 2026-03-09 19:10:03.818911 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.818915 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.818919 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818924 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.818928 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818932 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.818936 | controller | -- \' 2026-03-09 19:10:03.818941 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.818945 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.818949 | controller | - ' logger.go:42: 19:00:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.818954 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.818958 | controller | - ' logger.go:42: 19:00:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.818962 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.818967 | controller | found' 2026-03-09 19:10:03.818973 | controller | - ' logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.818978 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.818982 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.818986 | controller | - ' ' 2026-03-09 19:10:03.818991 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.818995 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.819001 | controller | - ' ' 2026-03-09 19:10:03.819005 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.819009 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.819014 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.819018 | controller | - ' exit 1' 2026-03-09 19:10:03.819022 | controller | - ' fi' 2026-03-09 19:10:03.819027 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.819031 | controller | - ' ' 2026-03-09 19:10:03.819035 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.819040 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.819044 | controller | | base64 -d)' 2026-03-09 19:10:03.819048 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.819053 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.819057 | controller | - ' exit 1' 2026-03-09 19:10:03.819061 | controller | - ' fi' 2026-03-09 19:10:03.819066 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.819070 | controller | - ' ' 2026-03-09 19:10:03.819074 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.819079 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.819083 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.819087 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.819091 | controller | --timeout=300s' 2026-03-09 19:10:03.819096 | controller | - ' ' 2026-03-09 19:10:03.819108 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.819113 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.819117 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.819122 | controller | - ' ' 2026-03-09 19:10:03.819126 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819131 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819135 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819139 | controller | - ' ' 2026-03-09 19:10:03.819143 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.819161 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.819166 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.819170 | controller | available"' 2026-03-09 19:10:03.819174 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.819179 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.819183 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.819187 | controller | - ' exit 1' 2026-03-09 19:10:03.819192 | controller | - ' fi' 2026-03-09 19:10:03.819196 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.819200 | controller | - ' ' 2026-03-09 19:10:03.819204 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.819209 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.819213 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819219 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.819223 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819228 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.819232 | controller | -- \' 2026-03-09 19:10:03.819236 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819241 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.819245 | controller | - ' logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.819249 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.819254 | controller | - ' logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.819258 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.819262 | controller | found' 2026-03-09 19:10:03.819267 | controller | - ' logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.819271 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.819275 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.819279 | controller | - ' ' 2026-03-09 19:10:03.819284 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.819288 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.819292 | controller | - ' ' 2026-03-09 19:10:03.819296 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.819301 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.819305 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.819309 | controller | - ' exit 1' 2026-03-09 19:10:03.819314 | controller | - ' fi' 2026-03-09 19:10:03.819320 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.819324 | controller | - ' ' 2026-03-09 19:10:03.819329 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.819333 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.819337 | controller | | base64 -d)' 2026-03-09 19:10:03.819342 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.819346 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.819350 | controller | - ' exit 1' 2026-03-09 19:10:03.819354 | controller | - ' fi' 2026-03-09 19:10:03.819359 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.819363 | controller | - ' ' 2026-03-09 19:10:03.819367 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.819372 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.819376 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.819380 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.819384 | controller | --timeout=300s' 2026-03-09 19:10:03.819389 | controller | - ' ' 2026-03-09 19:10:03.819393 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.819405 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.819409 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.819414 | controller | - ' ' 2026-03-09 19:10:03.819418 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819424 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819428 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819433 | controller | - ' ' 2026-03-09 19:10:03.819437 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.819441 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.819446 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.819450 | controller | available"' 2026-03-09 19:10:03.819454 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.819458 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.819463 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.819467 | controller | - ' exit 1' 2026-03-09 19:10:03.819471 | controller | - ' fi' 2026-03-09 19:10:03.819476 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.819480 | controller | - ' ' 2026-03-09 19:10:03.819484 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.819489 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.819493 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819497 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.819501 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819506 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.819510 | controller | -- \' 2026-03-09 19:10:03.819514 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819518 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.819525 | controller | - ' logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.819529 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.819533 | controller | - ' logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.819538 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.819542 | controller | found' 2026-03-09 19:10:03.819546 | controller | - ' logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.819550 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.819555 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.819559 | controller | - ' ' 2026-03-09 19:10:03.819565 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.819569 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.819573 | controller | - ' ' 2026-03-09 19:10:03.819577 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.819582 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.819586 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.819590 | controller | - ' exit 1' 2026-03-09 19:10:03.819595 | controller | - ' fi' 2026-03-09 19:10:03.819599 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.819603 | controller | - ' ' 2026-03-09 19:10:03.819607 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.819627 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.819632 | controller | | base64 -d)' 2026-03-09 19:10:03.819637 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.819641 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.819645 | controller | - ' exit 1' 2026-03-09 19:10:03.819649 | controller | - ' fi' 2026-03-09 19:10:03.819654 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.819658 | controller | - ' ' 2026-03-09 19:10:03.819662 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.819667 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.819671 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.819675 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.819679 | controller | --timeout=300s' 2026-03-09 19:10:03.819685 | controller | - ' ' 2026-03-09 19:10:03.819690 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.819702 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.819706 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.819710 | controller | - ' ' 2026-03-09 19:10:03.819715 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819719 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819723 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.819747 | controller | - ' ' 2026-03-09 19:10:03.819751 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.819756 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.819760 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.819764 | controller | available"' 2026-03-09 19:10:03.819768 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.819773 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.819777 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.819781 | controller | - ' exit 1' 2026-03-09 19:10:03.819789 | controller | - ' fi' 2026-03-09 19:10:03.819793 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.819797 | controller | - ' ' 2026-03-09 19:10:03.819801 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.819806 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.819810 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819814 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.819818 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819823 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.819827 | controller | -- \' 2026-03-09 19:10:03.819831 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.819837 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.819842 | controller | - ' logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.819846 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.819850 | controller | - ' logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.819854 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.819859 | controller | found' 2026-03-09 19:10:03.819863 | controller | - ' logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.819867 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.819872 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.819876 | controller | - ' ' 2026-03-09 19:10:03.819880 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.819884 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.819889 | controller | - ' ' 2026-03-09 19:10:03.819893 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.819897 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.819901 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.819906 | controller | - ' exit 1' 2026-03-09 19:10:03.819910 | controller | - ' fi' 2026-03-09 19:10:03.819914 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.819919 | controller | - ' ' 2026-03-09 19:10:03.819923 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.819927 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.819931 | controller | | base64 -d)' 2026-03-09 19:10:03.819936 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.819940 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.819944 | controller | - ' exit 1' 2026-03-09 19:10:03.819948 | controller | - ' fi' 2026-03-09 19:10:03.819953 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.819957 | controller | - ' ' 2026-03-09 19:10:03.819961 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.819965 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.819970 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.819974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.819978 | controller | --timeout=300s' 2026-03-09 19:10:03.819982 | controller | - ' ' 2026-03-09 19:10:03.819987 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.819998 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.820003 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.820007 | controller | - ' ' 2026-03-09 19:10:03.820011 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820016 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820020 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820024 | controller | - ' ' 2026-03-09 19:10:03.820028 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.820033 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.820040 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.820044 | controller | available"' 2026-03-09 19:10:03.820049 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.820055 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.820059 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.820063 | controller | - ' exit 1' 2026-03-09 19:10:03.820067 | controller | - ' fi' 2026-03-09 19:10:03.820072 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.820076 | controller | - ' ' 2026-03-09 19:10:03.820080 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.820084 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.820089 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820093 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.820097 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820101 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.820106 | controller | -- \' 2026-03-09 19:10:03.820110 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820114 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.820119 | controller | - ' logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.820123 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.820127 | controller | - ' logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.820132 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.820136 | controller | found' 2026-03-09 19:10:03.820140 | controller | - ' logger.go:42: 19:00:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.820167 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.820174 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.820179 | controller | - ' ' 2026-03-09 19:10:03.820183 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.820189 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.820194 | controller | - ' ' 2026-03-09 19:10:03.820198 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.820202 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.820206 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.820211 | controller | - ' exit 1' 2026-03-09 19:10:03.820215 | controller | - ' fi' 2026-03-09 19:10:03.820219 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.820223 | controller | - ' ' 2026-03-09 19:10:03.820228 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.820232 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.820236 | controller | | base64 -d)' 2026-03-09 19:10:03.820240 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.820245 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.820249 | controller | - ' exit 1' 2026-03-09 19:10:03.820253 | controller | - ' fi' 2026-03-09 19:10:03.820257 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.820264 | controller | - ' ' 2026-03-09 19:10:03.820268 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.820272 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.820276 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.820281 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.820285 | controller | --timeout=300s' 2026-03-09 19:10:03.820289 | controller | - ' ' 2026-03-09 19:10:03.820294 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.820298 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.820311 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.820315 | controller | - ' ' 2026-03-09 19:10:03.820319 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820324 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820328 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820332 | controller | - ' ' 2026-03-09 19:10:03.820337 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.820341 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.820345 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.820349 | controller | available"' 2026-03-09 19:10:03.820354 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.820358 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.820362 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.820366 | controller | - ' exit 1' 2026-03-09 19:10:03.820371 | controller | - ' fi' 2026-03-09 19:10:03.820377 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.820381 | controller | - ' ' 2026-03-09 19:10:03.820385 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.820389 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.820394 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820398 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.820402 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820406 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.820411 | controller | -- \' 2026-03-09 19:10:03.820415 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820419 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.820423 | controller | - ' logger.go:42: 19:00:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.820428 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.820432 | controller | - ' logger.go:42: 19:00:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.820436 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.820440 | controller | found' 2026-03-09 19:10:03.820446 | controller | - ' logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.820451 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.820455 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.820459 | controller | - ' ' 2026-03-09 19:10:03.820464 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.820468 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.820472 | controller | - ' ' 2026-03-09 19:10:03.820476 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.820481 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.820485 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.820489 | controller | - ' exit 1' 2026-03-09 19:10:03.820493 | controller | - ' fi' 2026-03-09 19:10:03.820498 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.820502 | controller | - ' ' 2026-03-09 19:10:03.820506 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.820510 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.820515 | controller | | base64 -d)' 2026-03-09 19:10:03.820519 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.820523 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.820527 | controller | - ' exit 1' 2026-03-09 19:10:03.820532 | controller | - ' fi' 2026-03-09 19:10:03.820536 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.820540 | controller | - ' ' 2026-03-09 19:10:03.820544 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.820549 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.820553 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.820557 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.820561 | controller | --timeout=300s' 2026-03-09 19:10:03.820566 | controller | - ' ' 2026-03-09 19:10:03.820570 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.820574 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.820585 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.820589 | controller | - ' ' 2026-03-09 19:10:03.820594 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820598 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820602 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820607 | controller | - ' ' 2026-03-09 19:10:03.820611 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.820615 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.820619 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.820624 | controller | available"' 2026-03-09 19:10:03.820628 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.820632 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.820636 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.820641 | controller | - ' exit 1' 2026-03-09 19:10:03.820645 | controller | - ' fi' 2026-03-09 19:10:03.820651 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.820655 | controller | - ' ' 2026-03-09 19:10:03.820659 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.820664 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.820668 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820672 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.820676 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820681 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.820685 | controller | -- \' 2026-03-09 19:10:03.820689 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820693 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.820699 | controller | - ' logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.820704 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.820708 | controller | - ' logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.820712 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.820716 | controller | found' 2026-03-09 19:10:03.820721 | controller | - ' logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.820725 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.820729 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.820733 | controller | - ' ' 2026-03-09 19:10:03.820738 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.820742 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.820746 | controller | - ' ' 2026-03-09 19:10:03.820750 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.820756 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.820760 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.820765 | controller | - ' exit 1' 2026-03-09 19:10:03.820769 | controller | - ' fi' 2026-03-09 19:10:03.820773 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.820778 | controller | - ' ' 2026-03-09 19:10:03.820782 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.820786 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.820790 | controller | | base64 -d)' 2026-03-09 19:10:03.820795 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.820799 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.820803 | controller | - ' exit 1' 2026-03-09 19:10:03.820807 | controller | - ' fi' 2026-03-09 19:10:03.820812 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.820816 | controller | - ' ' 2026-03-09 19:10:03.820820 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.820825 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.820829 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.820833 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.820837 | controller | --timeout=300s' 2026-03-09 19:10:03.820842 | controller | - ' ' 2026-03-09 19:10:03.820848 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.820853 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.820857 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.820861 | controller | - ' ' 2026-03-09 19:10:03.820874 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820878 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820882 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.820887 | controller | - ' ' 2026-03-09 19:10:03.820891 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.820895 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.820900 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.820904 | controller | available"' 2026-03-09 19:10:03.820908 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.820912 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.820917 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.820921 | controller | - ' exit 1' 2026-03-09 19:10:03.820925 | controller | - ' fi' 2026-03-09 19:10:03.820929 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.820934 | controller | - ' ' 2026-03-09 19:10:03.820938 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.820942 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.820946 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820951 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.820955 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820959 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.820963 | controller | -- \' 2026-03-09 19:10:03.820967 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.820972 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.820976 | controller | - ' logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.820980 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.820985 | controller | - ' logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.820989 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.820993 | controller | found' 2026-03-09 19:10:03.820997 | controller | - ' logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.821001 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.821007 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.821013 | controller | - ' ' 2026-03-09 19:10:03.821017 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.821022 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.821026 | controller | - ' ' 2026-03-09 19:10:03.821030 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.821037 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.821042 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.821046 | controller | - ' exit 1' 2026-03-09 19:10:03.821051 | controller | - ' fi' 2026-03-09 19:10:03.821055 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.821059 | controller | - ' ' 2026-03-09 19:10:03.821063 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.821068 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.821072 | controller | | base64 -d)' 2026-03-09 19:10:03.821076 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.821081 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.821085 | controller | - ' exit 1' 2026-03-09 19:10:03.821089 | controller | - ' fi' 2026-03-09 19:10:03.821093 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.821098 | controller | - ' ' 2026-03-09 19:10:03.821102 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.821106 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.821110 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.821115 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.821119 | controller | --timeout=300s' 2026-03-09 19:10:03.821123 | controller | - ' ' 2026-03-09 19:10:03.821128 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.821132 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.821136 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.821140 | controller | - ' ' 2026-03-09 19:10:03.821162 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821169 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821173 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821177 | controller | - ' ' 2026-03-09 19:10:03.821182 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.821186 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.821191 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.821195 | controller | available"' 2026-03-09 19:10:03.821199 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.821203 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.821208 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.821212 | controller | - ' exit 1' 2026-03-09 19:10:03.821216 | controller | - ' fi' 2026-03-09 19:10:03.821221 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.821225 | controller | - ' ' 2026-03-09 19:10:03.821229 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.821233 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.821238 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821242 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.821249 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821253 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.821257 | controller | -- \' 2026-03-09 19:10:03.821262 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821266 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.821270 | controller | - ' logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.821275 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.821279 | controller | - ' logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.821283 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.821288 | controller | found' 2026-03-09 19:10:03.821294 | controller | - ' logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.821298 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.821302 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.821307 | controller | - ' ' 2026-03-09 19:10:03.821311 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.821315 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.821319 | controller | - ' ' 2026-03-09 19:10:03.821324 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.821328 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.821332 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.821336 | controller | - ' exit 1' 2026-03-09 19:10:03.821341 | controller | - ' fi' 2026-03-09 19:10:03.821345 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.821352 | controller | - ' ' 2026-03-09 19:10:03.821357 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.821361 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.821365 | controller | | base64 -d)' 2026-03-09 19:10:03.821369 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.821375 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.821380 | controller | - ' exit 1' 2026-03-09 19:10:03.821384 | controller | - ' fi' 2026-03-09 19:10:03.821388 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.821392 | controller | - ' ' 2026-03-09 19:10:03.821397 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.821401 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.821405 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.821409 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.821414 | controller | --timeout=300s' 2026-03-09 19:10:03.821418 | controller | - ' ' 2026-03-09 19:10:03.821424 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.821428 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.821433 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.821437 | controller | - ' ' 2026-03-09 19:10:03.821441 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821455 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821460 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821464 | controller | - ' ' 2026-03-09 19:10:03.821469 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.821473 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.821477 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.821481 | controller | available"' 2026-03-09 19:10:03.821486 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.821490 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.821494 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.821498 | controller | - ' exit 1' 2026-03-09 19:10:03.821503 | controller | - ' fi' 2026-03-09 19:10:03.821507 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.821511 | controller | - ' ' 2026-03-09 19:10:03.821515 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.821520 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.821524 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821528 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.821532 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821537 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.821541 | controller | -- \' 2026-03-09 19:10:03.821545 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821550 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.821554 | controller | - ' logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.821558 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.821563 | controller | - ' logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.821567 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.821571 | controller | found' 2026-03-09 19:10:03.821575 | controller | - ' logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.821580 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.821584 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.821588 | controller | - ' ' 2026-03-09 19:10:03.821592 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.821597 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.821601 | controller | - ' ' 2026-03-09 19:10:03.821605 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.821609 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.821613 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.821618 | controller | - ' exit 1' 2026-03-09 19:10:03.821622 | controller | - ' fi' 2026-03-09 19:10:03.821626 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.821630 | controller | - ' ' 2026-03-09 19:10:03.821635 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.821639 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.821646 | controller | | base64 -d)' 2026-03-09 19:10:03.821650 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.821654 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.821659 | controller | - ' exit 1' 2026-03-09 19:10:03.821663 | controller | - ' fi' 2026-03-09 19:10:03.821667 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.821672 | controller | - ' ' 2026-03-09 19:10:03.821676 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.821680 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.821684 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.821689 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.821693 | controller | --timeout=300s' 2026-03-09 19:10:03.821697 | controller | - ' ' 2026-03-09 19:10:03.821701 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.821706 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.821710 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.821714 | controller | - ' ' 2026-03-09 19:10:03.821718 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821729 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821734 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.821738 | controller | - ' ' 2026-03-09 19:10:03.821742 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.821748 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.821753 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.821757 | controller | available"' 2026-03-09 19:10:03.821761 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.821765 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.821770 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.821774 | controller | - ' exit 1' 2026-03-09 19:10:03.821778 | controller | - ' fi' 2026-03-09 19:10:03.821782 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.821787 | controller | - ' ' 2026-03-09 19:10:03.821791 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.821795 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.821800 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821804 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.821808 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821813 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.821817 | controller | -- \' 2026-03-09 19:10:03.821822 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.821826 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.821832 | controller | - ' logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.821836 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.821841 | controller | - ' logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.821845 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.821849 | controller | found' 2026-03-09 19:10:03.821854 | controller | - ' logger.go:42: 19:00:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.821858 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.821862 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.821867 | controller | - ' ' 2026-03-09 19:10:03.821871 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.821875 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.821880 | controller | - ' ' 2026-03-09 19:10:03.821884 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.821888 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.821892 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.821897 | controller | - ' exit 1' 2026-03-09 19:10:03.821901 | controller | - ' fi' 2026-03-09 19:10:03.821905 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.821909 | controller | - ' ' 2026-03-09 19:10:03.821914 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.821918 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.821922 | controller | | base64 -d)' 2026-03-09 19:10:03.821927 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.821931 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.821935 | controller | - ' exit 1' 2026-03-09 19:10:03.821939 | controller | - ' fi' 2026-03-09 19:10:03.821944 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.821948 | controller | - ' ' 2026-03-09 19:10:03.821952 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.821957 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.821962 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.821967 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.821971 | controller | --timeout=300s' 2026-03-09 19:10:03.821975 | controller | - ' ' 2026-03-09 19:10:03.821980 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.821984 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.821988 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.821992 | controller | - ' ' 2026-03-09 19:10:03.821997 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822001 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822042 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822048 | controller | - ' ' 2026-03-09 19:10:03.822053 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.822057 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.822061 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.822068 | controller | available"' 2026-03-09 19:10:03.822073 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.822077 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.822081 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.822085 | controller | - ' exit 1' 2026-03-09 19:10:03.822090 | controller | - ' fi' 2026-03-09 19:10:03.822094 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.822098 | controller | - ' ' 2026-03-09 19:10:03.822102 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.822107 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.822111 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822115 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.822119 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822124 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.822128 | controller | -- \' 2026-03-09 19:10:03.822132 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822137 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.822141 | controller | - ' logger.go:42: 19:00:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.822155 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.822162 | controller | - ' logger.go:42: 19:00:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.822166 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.822171 | controller | found' 2026-03-09 19:10:03.822175 | controller | - ' logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.822179 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.822183 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.822188 | controller | - ' ' 2026-03-09 19:10:03.822192 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.822196 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.822201 | controller | - ' ' 2026-03-09 19:10:03.822205 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.822209 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.822214 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.822218 | controller | - ' exit 1' 2026-03-09 19:10:03.822222 | controller | - ' fi' 2026-03-09 19:10:03.822226 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.822231 | controller | - ' ' 2026-03-09 19:10:03.822235 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.822239 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.822243 | controller | | base64 -d)' 2026-03-09 19:10:03.822248 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.822254 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.822259 | controller | - ' exit 1' 2026-03-09 19:10:03.822263 | controller | - ' fi' 2026-03-09 19:10:03.822267 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.822271 | controller | - ' ' 2026-03-09 19:10:03.822277 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.822282 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.822286 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.822290 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.822294 | controller | --timeout=300s' 2026-03-09 19:10:03.822299 | controller | - ' ' 2026-03-09 19:10:03.822303 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.822307 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.822311 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.822316 | controller | - ' ' 2026-03-09 19:10:03.822320 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822324 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822336 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822341 | controller | - ' ' 2026-03-09 19:10:03.822345 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.822350 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.822356 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.822361 | controller | available"' 2026-03-09 19:10:03.822367 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.822373 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.822378 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.822384 | controller | - ' exit 1' 2026-03-09 19:10:03.822389 | controller | - ' fi' 2026-03-09 19:10:03.822395 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.822401 | controller | - ' ' 2026-03-09 19:10:03.822407 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.822412 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.822417 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822423 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.822428 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822434 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.822439 | controller | -- \' 2026-03-09 19:10:03.822445 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822451 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.822457 | controller | - ' logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.822462 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.822466 | controller | - ' logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.822470 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.822474 | controller | found' 2026-03-09 19:10:03.822479 | controller | - ' logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.822487 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.822493 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.822499 | controller | - ' ' 2026-03-09 19:10:03.822505 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.822511 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.822517 | controller | - ' ' 2026-03-09 19:10:03.822522 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.822528 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.822533 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.822539 | controller | - ' exit 1' 2026-03-09 19:10:03.822545 | controller | - ' fi' 2026-03-09 19:10:03.822550 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.822555 | controller | - ' ' 2026-03-09 19:10:03.822561 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.822567 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.822573 | controller | | base64 -d)' 2026-03-09 19:10:03.822583 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.822588 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.822592 | controller | - ' exit 1' 2026-03-09 19:10:03.822597 | controller | - ' fi' 2026-03-09 19:10:03.822601 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.822605 | controller | - ' ' 2026-03-09 19:10:03.822610 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.822614 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.822618 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.822624 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.822628 | controller | --timeout=300s' 2026-03-09 19:10:03.822633 | controller | - ' ' 2026-03-09 19:10:03.822637 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.822641 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.822645 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.822650 | controller | - ' ' 2026-03-09 19:10:03.822654 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822658 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822662 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822676 | controller | - ' ' 2026-03-09 19:10:03.822681 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.822685 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.822690 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.822694 | controller | available"' 2026-03-09 19:10:03.822698 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.822704 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.822708 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.822713 | controller | - ' exit 1' 2026-03-09 19:10:03.822717 | controller | - ' fi' 2026-03-09 19:10:03.822721 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.822730 | controller | - ' ' 2026-03-09 19:10:03.822734 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.822739 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.822743 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822747 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.822751 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822756 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.822760 | controller | -- \' 2026-03-09 19:10:03.822764 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.822769 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.822773 | controller | - ' logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.822777 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.822782 | controller | - ' logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.822786 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.822790 | controller | found' 2026-03-09 19:10:03.822795 | controller | - ' logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.822799 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.822803 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.822807 | controller | - ' ' 2026-03-09 19:10:03.822812 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.822816 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.822820 | controller | - ' ' 2026-03-09 19:10:03.822825 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.822829 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.822833 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.822837 | controller | - ' exit 1' 2026-03-09 19:10:03.822842 | controller | - ' fi' 2026-03-09 19:10:03.822846 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.822850 | controller | - ' ' 2026-03-09 19:10:03.822855 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.822859 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.822863 | controller | | base64 -d)' 2026-03-09 19:10:03.822867 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.822872 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.822876 | controller | - ' exit 1' 2026-03-09 19:10:03.822880 | controller | - ' fi' 2026-03-09 19:10:03.822885 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.822889 | controller | - ' ' 2026-03-09 19:10:03.822893 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.822897 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.822902 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.822906 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.822910 | controller | --timeout=300s' 2026-03-09 19:10:03.822915 | controller | - ' ' 2026-03-09 19:10:03.822919 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.822925 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.822931 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.822935 | controller | - ' ' 2026-03-09 19:10:03.822939 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822943 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822948 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.822952 | controller | - ' ' 2026-03-09 19:10:03.822964 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.822968 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.822973 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.822977 | controller | available"' 2026-03-09 19:10:03.822982 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.822986 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.822990 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.822994 | controller | - ' exit 1' 2026-03-09 19:10:03.822999 | controller | - ' fi' 2026-03-09 19:10:03.823003 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.823007 | controller | - ' ' 2026-03-09 19:10:03.823012 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.823016 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.823020 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823024 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.823029 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823033 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.823037 | controller | -- \' 2026-03-09 19:10:03.823042 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823046 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.823050 | controller | - ' logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.823055 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.823059 | controller | - ' logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.823063 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.823067 | controller | found' 2026-03-09 19:10:03.823072 | controller | - ' logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.823076 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.823080 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.823085 | controller | - ' ' 2026-03-09 19:10:03.823089 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.823093 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.823098 | controller | - ' ' 2026-03-09 19:10:03.823102 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.823108 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.823112 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.823117 | controller | - ' exit 1' 2026-03-09 19:10:03.823121 | controller | - ' fi' 2026-03-09 19:10:03.823125 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.823130 | controller | - ' ' 2026-03-09 19:10:03.823134 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.823138 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.823143 | controller | | base64 -d)' 2026-03-09 19:10:03.823161 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.823166 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.823170 | controller | - ' exit 1' 2026-03-09 19:10:03.823174 | controller | - ' fi' 2026-03-09 19:10:03.823179 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.823183 | controller | - ' ' 2026-03-09 19:10:03.823187 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.823192 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.823196 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.823200 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.823204 | controller | --timeout=300s' 2026-03-09 19:10:03.823209 | controller | - ' ' 2026-03-09 19:10:03.823213 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.823217 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.823222 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.823226 | controller | - ' ' 2026-03-09 19:10:03.823230 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823235 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823239 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823243 | controller | - ' ' 2026-03-09 19:10:03.823249 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.823264 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.823269 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.823273 | controller | available"' 2026-03-09 19:10:03.823277 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.823282 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.823286 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.823290 | controller | - ' exit 1' 2026-03-09 19:10:03.823295 | controller | - ' fi' 2026-03-09 19:10:03.823299 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.823303 | controller | - ' ' 2026-03-09 19:10:03.823308 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.823312 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.823316 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823320 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.823327 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823331 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.823335 | controller | -- \' 2026-03-09 19:10:03.823340 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823344 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.823348 | controller | - ' logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.823353 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.823357 | controller | - ' logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.823361 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.823366 | controller | found' 2026-03-09 19:10:03.823370 | controller | - ' logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.823374 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.823378 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.823383 | controller | - ' ' 2026-03-09 19:10:03.823387 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.823391 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.823396 | controller | - ' ' 2026-03-09 19:10:03.823400 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.823404 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.823408 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.823413 | controller | - ' exit 1' 2026-03-09 19:10:03.823417 | controller | - ' fi' 2026-03-09 19:10:03.823421 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.823426 | controller | - ' ' 2026-03-09 19:10:03.823430 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.823434 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.823438 | controller | | base64 -d)' 2026-03-09 19:10:03.823443 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.823449 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.823453 | controller | - ' exit 1' 2026-03-09 19:10:03.823457 | controller | - ' fi' 2026-03-09 19:10:03.823461 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.823466 | controller | - ' ' 2026-03-09 19:10:03.823470 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.823474 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.823479 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.823483 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.823487 | controller | --timeout=300s' 2026-03-09 19:10:03.823491 | controller | - ' ' 2026-03-09 19:10:03.823496 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.823500 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.823504 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.823509 | controller | - ' ' 2026-03-09 19:10:03.823513 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823517 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823523 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823527 | controller | - ' ' 2026-03-09 19:10:03.823532 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.823543 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.823548 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.823552 | controller | available"' 2026-03-09 19:10:03.823556 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.823561 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.823565 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.823569 | controller | - ' exit 1' 2026-03-09 19:10:03.823574 | controller | - ' fi' 2026-03-09 19:10:03.823578 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.823582 | controller | - ' ' 2026-03-09 19:10:03.823586 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.823591 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.823595 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823599 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.823604 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823624 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.823630 | controller | -- \' 2026-03-09 19:10:03.823635 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823640 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.823646 | controller | - ' logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.823650 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.823655 | controller | - ' logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.823659 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.823663 | controller | found' 2026-03-09 19:10:03.823668 | controller | - ' logger.go:42: 19:00:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.823672 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.823676 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.823680 | controller | - ' ' 2026-03-09 19:10:03.823686 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.823690 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.823695 | controller | - ' ' 2026-03-09 19:10:03.823715 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.823719 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.823724 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.823728 | controller | - ' exit 1' 2026-03-09 19:10:03.823732 | controller | - ' fi' 2026-03-09 19:10:03.823737 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.823741 | controller | - ' ' 2026-03-09 19:10:03.823745 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.823749 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.823756 | controller | | base64 -d)' 2026-03-09 19:10:03.823760 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.823764 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.823769 | controller | - ' exit 1' 2026-03-09 19:10:03.823773 | controller | - ' fi' 2026-03-09 19:10:03.823777 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.823782 | controller | - ' ' 2026-03-09 19:10:03.823786 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.823790 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.823794 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.823799 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.823803 | controller | --timeout=300s' 2026-03-09 19:10:03.823809 | controller | - ' ' 2026-03-09 19:10:03.823814 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.823818 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.823822 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.823827 | controller | - ' ' 2026-03-09 19:10:03.823831 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823835 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823839 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.823844 | controller | - ' ' 2026-03-09 19:10:03.823848 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.823852 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.823866 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.823871 | controller | available"' 2026-03-09 19:10:03.823875 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.823880 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.823884 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.823888 | controller | - ' exit 1' 2026-03-09 19:10:03.823893 | controller | - ' fi' 2026-03-09 19:10:03.823897 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.823901 | controller | - ' ' 2026-03-09 19:10:03.823906 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.823910 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.823914 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823919 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.823923 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823927 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.823931 | controller | -- \' 2026-03-09 19:10:03.823936 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.823940 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.823944 | controller | - ' logger.go:42: 19:00:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.823950 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.823956 | controller | - ' logger.go:42: 19:00:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.823961 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.823965 | controller | found' 2026-03-09 19:10:03.823969 | controller | - ' logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.823973 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.823978 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.823982 | controller | - ' ' 2026-03-09 19:10:03.823986 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.823991 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.823995 | controller | - ' ' 2026-03-09 19:10:03.823999 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.824003 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.824008 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.824012 | controller | - ' exit 1' 2026-03-09 19:10:03.824016 | controller | - ' fi' 2026-03-09 19:10:03.824021 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.824025 | controller | - ' ' 2026-03-09 19:10:03.824033 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.824037 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.824041 | controller | | base64 -d)' 2026-03-09 19:10:03.824046 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.824050 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.824054 | controller | - ' exit 1' 2026-03-09 19:10:03.824058 | controller | - ' fi' 2026-03-09 19:10:03.824063 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.824067 | controller | - ' ' 2026-03-09 19:10:03.824071 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.824076 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.824080 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.824084 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.824088 | controller | --timeout=300s' 2026-03-09 19:10:03.824093 | controller | - ' ' 2026-03-09 19:10:03.824097 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.824101 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.824106 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.824110 | controller | - ' ' 2026-03-09 19:10:03.824114 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824119 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824123 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824127 | controller | - ' ' 2026-03-09 19:10:03.824131 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.824136 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.824163 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.824172 | controller | available"' 2026-03-09 19:10:03.824177 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.824181 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.824185 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.824190 | controller | - ' exit 1' 2026-03-09 19:10:03.824194 | controller | - ' fi' 2026-03-09 19:10:03.824198 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.824202 | controller | - ' ' 2026-03-09 19:10:03.824207 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.824211 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.824215 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824219 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.824224 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824228 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.824232 | controller | -- \' 2026-03-09 19:10:03.824237 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824241 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.824245 | controller | - ' logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.824250 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.824254 | controller | - ' logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.824258 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.824262 | controller | found' 2026-03-09 19:10:03.824267 | controller | - ' logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.824271 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.824275 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.824280 | controller | - ' ' 2026-03-09 19:10:03.824284 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.824288 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.824292 | controller | - ' ' 2026-03-09 19:10:03.824297 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.824301 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.824305 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.824310 | controller | - ' exit 1' 2026-03-09 19:10:03.824314 | controller | - ' fi' 2026-03-09 19:10:03.824318 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.824322 | controller | - ' ' 2026-03-09 19:10:03.824327 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.824331 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.824335 | controller | | base64 -d)' 2026-03-09 19:10:03.824340 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.824344 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.824348 | controller | - ' exit 1' 2026-03-09 19:10:03.824352 | controller | - ' fi' 2026-03-09 19:10:03.824357 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.824361 | controller | - ' ' 2026-03-09 19:10:03.824365 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.824378 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.824382 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.824387 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.824391 | controller | --timeout=300s' 2026-03-09 19:10:03.824395 | controller | - ' ' 2026-03-09 19:10:03.824400 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.824404 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.824408 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.824412 | controller | - ' ' 2026-03-09 19:10:03.824417 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824421 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824425 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824429 | controller | - ' ' 2026-03-09 19:10:03.824434 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.824438 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.824442 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.824446 | controller | available"' 2026-03-09 19:10:03.824460 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.824465 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.824469 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.824474 | controller | - ' exit 1' 2026-03-09 19:10:03.824478 | controller | - ' fi' 2026-03-09 19:10:03.824482 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.824487 | controller | - ' ' 2026-03-09 19:10:03.824491 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.824495 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.824500 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824504 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.824508 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824512 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.824517 | controller | -- \' 2026-03-09 19:10:03.824521 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824525 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.824529 | controller | - ' logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.824534 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.824538 | controller | - ' logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.824542 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.824547 | controller | found' 2026-03-09 19:10:03.824551 | controller | - ' logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.824555 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.824561 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.824565 | controller | - ' ' 2026-03-09 19:10:03.824570 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.824574 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.824578 | controller | - ' ' 2026-03-09 19:10:03.824583 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.824587 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.824591 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.824595 | controller | - ' exit 1' 2026-03-09 19:10:03.824600 | controller | - ' fi' 2026-03-09 19:10:03.824604 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.824608 | controller | - ' ' 2026-03-09 19:10:03.824613 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.824617 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.824621 | controller | | base64 -d)' 2026-03-09 19:10:03.824625 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.824630 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.824634 | controller | - ' exit 1' 2026-03-09 19:10:03.824638 | controller | - ' fi' 2026-03-09 19:10:03.824643 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.824647 | controller | - ' ' 2026-03-09 19:10:03.824651 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.824655 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.824660 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.824664 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.824668 | controller | --timeout=300s' 2026-03-09 19:10:03.824672 | controller | - ' ' 2026-03-09 19:10:03.824677 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.824681 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.824685 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.824689 | controller | - ' ' 2026-03-09 19:10:03.824694 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824698 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824702 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824707 | controller | - ' ' 2026-03-09 19:10:03.824713 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.824717 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.824721 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.824726 | controller | available"' 2026-03-09 19:10:03.824730 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.824741 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.824746 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.824750 | controller | - ' exit 1' 2026-03-09 19:10:03.824754 | controller | - ' fi' 2026-03-09 19:10:03.824759 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.824765 | controller | - ' ' 2026-03-09 19:10:03.824769 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.824773 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.824778 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824782 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.824786 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824790 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.824795 | controller | -- \' 2026-03-09 19:10:03.824799 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.824803 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.824808 | controller | - ' logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.824812 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.824818 | controller | - ' logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.824822 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.824826 | controller | found' 2026-03-09 19:10:03.824831 | controller | - ' logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.824835 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.824840 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.824844 | controller | - ' ' 2026-03-09 19:10:03.824848 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.824853 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.824857 | controller | - ' ' 2026-03-09 19:10:03.824861 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.824865 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.824870 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.824874 | controller | - ' exit 1' 2026-03-09 19:10:03.824878 | controller | - ' fi' 2026-03-09 19:10:03.824882 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.824887 | controller | - ' ' 2026-03-09 19:10:03.824891 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.824895 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.824899 | controller | | base64 -d)' 2026-03-09 19:10:03.824904 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.824908 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.824912 | controller | - ' exit 1' 2026-03-09 19:10:03.824916 | controller | - ' fi' 2026-03-09 19:10:03.824921 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.824925 | controller | - ' ' 2026-03-09 19:10:03.824929 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.824933 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.824938 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.824942 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.824946 | controller | --timeout=300s' 2026-03-09 19:10:03.824950 | controller | - ' ' 2026-03-09 19:10:03.824955 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.824961 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.824965 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.824969 | controller | - ' ' 2026-03-09 19:10:03.824975 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824980 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824985 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.824989 | controller | - ' ' 2026-03-09 19:10:03.824993 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.824998 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.825002 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.825006 | controller | available"' 2026-03-09 19:10:03.825010 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.825022 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.825026 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.825030 | controller | - ' exit 1' 2026-03-09 19:10:03.825035 | controller | - ' fi' 2026-03-09 19:10:03.825039 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.825043 | controller | - ' ' 2026-03-09 19:10:03.825079 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.825086 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.825090 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825094 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.825099 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825103 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.825107 | controller | -- \' 2026-03-09 19:10:03.825112 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825116 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.825120 | controller | - ' logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.825125 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.825129 | controller | - ' logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.825133 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.825137 | controller | found' 2026-03-09 19:10:03.825142 | controller | - ' logger.go:42: 19:00:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.825158 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.825165 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.825169 | controller | - ' ' 2026-03-09 19:10:03.825173 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.825178 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.825182 | controller | - ' ' 2026-03-09 19:10:03.825186 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.825190 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.825197 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.825201 | controller | - ' exit 1' 2026-03-09 19:10:03.825205 | controller | - ' fi' 2026-03-09 19:10:03.825233 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.825238 | controller | - ' ' 2026-03-09 19:10:03.825243 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.825247 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.825251 | controller | | base64 -d)' 2026-03-09 19:10:03.825256 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.825260 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.825264 | controller | - ' exit 1' 2026-03-09 19:10:03.825268 | controller | - ' fi' 2026-03-09 19:10:03.825273 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.825277 | controller | - ' ' 2026-03-09 19:10:03.825281 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.825286 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.825290 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.825294 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.825298 | controller | --timeout=300s' 2026-03-09 19:10:03.825303 | controller | - ' ' 2026-03-09 19:10:03.825307 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.825311 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.825316 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.825320 | controller | - ' ' 2026-03-09 19:10:03.825324 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825328 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825333 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825337 | controller | - ' ' 2026-03-09 19:10:03.825341 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.825345 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.825350 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.825354 | controller | available"' 2026-03-09 19:10:03.825358 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.825365 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.825378 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.825382 | controller | - ' exit 1' 2026-03-09 19:10:03.825386 | controller | - ' fi' 2026-03-09 19:10:03.825391 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.825395 | controller | - ' ' 2026-03-09 19:10:03.825399 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.825404 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.825408 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825412 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.825416 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825424 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.825428 | controller | -- \' 2026-03-09 19:10:03.825432 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825436 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.825441 | controller | - ' logger.go:42: 19:00:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.825445 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.825449 | controller | - ' logger.go:42: 19:00:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.825453 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.825458 | controller | found' 2026-03-09 19:10:03.825462 | controller | - ' logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.825466 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.825470 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.825475 | controller | - ' ' 2026-03-09 19:10:03.825479 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.825483 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.825487 | controller | - ' ' 2026-03-09 19:10:03.825492 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.825496 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.825500 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.825505 | controller | - ' exit 1' 2026-03-09 19:10:03.825509 | controller | - ' fi' 2026-03-09 19:10:03.825513 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.825517 | controller | - ' ' 2026-03-09 19:10:03.825522 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.825526 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.825530 | controller | | base64 -d)' 2026-03-09 19:10:03.825534 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.825539 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.825543 | controller | - ' exit 1' 2026-03-09 19:10:03.825547 | controller | - ' fi' 2026-03-09 19:10:03.825552 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.825556 | controller | - ' ' 2026-03-09 19:10:03.825560 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.825564 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.825569 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.825573 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.825577 | controller | --timeout=300s' 2026-03-09 19:10:03.825581 | controller | - ' ' 2026-03-09 19:10:03.825586 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.825590 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.825594 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.825598 | controller | - ' ' 2026-03-09 19:10:03.825603 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825607 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825614 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825618 | controller | - ' ' 2026-03-09 19:10:03.825623 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.825627 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.825631 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.825636 | controller | available"' 2026-03-09 19:10:03.825640 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.825644 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.825648 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.825653 | controller | - ' exit 1' 2026-03-09 19:10:03.825663 | controller | - ' fi' 2026-03-09 19:10:03.825668 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.825672 | controller | - ' ' 2026-03-09 19:10:03.825677 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.825681 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.825685 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825691 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.825695 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825700 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.825704 | controller | -- \' 2026-03-09 19:10:03.825708 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825712 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.825717 | controller | - ' logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.825721 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.825725 | controller | - ' logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.825729 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.825733 | controller | found' 2026-03-09 19:10:03.825738 | controller | - ' logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.825742 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.825746 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.825750 | controller | - ' ' 2026-03-09 19:10:03.825755 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.825759 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.825763 | controller | - ' ' 2026-03-09 19:10:03.825767 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.825772 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.825776 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.825780 | controller | - ' exit 1' 2026-03-09 19:10:03.825784 | controller | - ' fi' 2026-03-09 19:10:03.825789 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.825793 | controller | - ' ' 2026-03-09 19:10:03.825797 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.825801 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.825806 | controller | | base64 -d)' 2026-03-09 19:10:03.825812 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.825816 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.825820 | controller | - ' exit 1' 2026-03-09 19:10:03.825825 | controller | - ' fi' 2026-03-09 19:10:03.825831 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.825835 | controller | - ' ' 2026-03-09 19:10:03.825839 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.825843 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.825848 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.825852 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.825856 | controller | --timeout=300s' 2026-03-09 19:10:03.825861 | controller | - ' ' 2026-03-09 19:10:03.825865 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.825869 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.825873 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.825878 | controller | - ' ' 2026-03-09 19:10:03.825882 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825886 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825890 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.825895 | controller | - ' ' 2026-03-09 19:10:03.825899 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.825903 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.825907 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.825912 | controller | available"' 2026-03-09 19:10:03.825916 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.825920 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.825924 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.825929 | controller | - ' exit 1' 2026-03-09 19:10:03.825933 | controller | - ' fi' 2026-03-09 19:10:03.825944 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.825948 | controller | - ' ' 2026-03-09 19:10:03.825953 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.825957 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.825961 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825966 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.825970 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825974 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.825980 | controller | -- \' 2026-03-09 19:10:03.825984 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.825989 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.825993 | controller | - ' logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.825999 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.826003 | controller | - ' logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.826007 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.826012 | controller | found' 2026-03-09 19:10:03.826016 | controller | - ' logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.826020 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.826024 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.826029 | controller | - ' ' 2026-03-09 19:10:03.826033 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.826037 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.826042 | controller | - ' ' 2026-03-09 19:10:03.826046 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.826050 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.826055 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.826059 | controller | - ' exit 1' 2026-03-09 19:10:03.826063 | controller | - ' fi' 2026-03-09 19:10:03.826067 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.826072 | controller | - ' ' 2026-03-09 19:10:03.826076 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.826080 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.826084 | controller | | base64 -d)' 2026-03-09 19:10:03.826089 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.826093 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.826097 | controller | - ' exit 1' 2026-03-09 19:10:03.826102 | controller | - ' fi' 2026-03-09 19:10:03.826106 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.826110 | controller | - ' ' 2026-03-09 19:10:03.826114 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.826119 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.826123 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.826127 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.826131 | controller | --timeout=300s' 2026-03-09 19:10:03.826136 | controller | - ' ' 2026-03-09 19:10:03.826140 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.826156 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.826163 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.826167 | controller | - ' ' 2026-03-09 19:10:03.826171 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826176 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826180 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826184 | controller | - ' ' 2026-03-09 19:10:03.826189 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.826193 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.826197 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.826201 | controller | available"' 2026-03-09 19:10:03.826208 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.826212 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.826216 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.826220 | controller | - ' exit 1' 2026-03-09 19:10:03.826225 | controller | - ' fi' 2026-03-09 19:10:03.826236 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.826241 | controller | - ' ' 2026-03-09 19:10:03.826245 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.826250 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.826254 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826258 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.826262 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826267 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.826271 | controller | -- \' 2026-03-09 19:10:03.826275 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826281 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.826286 | controller | - ' logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.826290 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.826294 | controller | - ' logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.826299 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.826303 | controller | found' 2026-03-09 19:10:03.826307 | controller | - ' logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.826311 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.826316 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.826320 | controller | - ' ' 2026-03-09 19:10:03.826324 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.826328 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.826333 | controller | - ' ' 2026-03-09 19:10:03.826337 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.826341 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.826346 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.826350 | controller | - ' exit 1' 2026-03-09 19:10:03.826354 | controller | - ' fi' 2026-03-09 19:10:03.826358 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.826363 | controller | - ' ' 2026-03-09 19:10:03.826367 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.826371 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.826375 | controller | | base64 -d)' 2026-03-09 19:10:03.826380 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.826385 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.826390 | controller | - ' exit 1' 2026-03-09 19:10:03.826394 | controller | - ' fi' 2026-03-09 19:10:03.826398 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.826403 | controller | - ' ' 2026-03-09 19:10:03.826407 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.826411 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.826417 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.826422 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.826426 | controller | --timeout=300s' 2026-03-09 19:10:03.826430 | controller | - ' ' 2026-03-09 19:10:03.826434 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.826439 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.826443 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.826447 | controller | - ' ' 2026-03-09 19:10:03.826451 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826456 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826460 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826464 | controller | - ' ' 2026-03-09 19:10:03.826469 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.826473 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.826477 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.826481 | controller | available"' 2026-03-09 19:10:03.826486 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.826490 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.826494 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.826498 | controller | - ' exit 1' 2026-03-09 19:10:03.826503 | controller | - ' fi' 2026-03-09 19:10:03.826507 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.826511 | controller | - ' ' 2026-03-09 19:10:03.826522 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.826526 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.826531 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826535 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.826539 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826543 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.826548 | controller | -- \' 2026-03-09 19:10:03.826552 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826556 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.826560 | controller | - ' logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.826565 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.826569 | controller | - ' logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.826573 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.826578 | controller | found' 2026-03-09 19:10:03.826582 | controller | - ' logger.go:42: 19:00:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.826586 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.826590 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.826596 | controller | - ' ' 2026-03-09 19:10:03.826601 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.826605 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.826609 | controller | - ' ' 2026-03-09 19:10:03.826613 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.826618 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.826622 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.826626 | controller | - ' exit 1' 2026-03-09 19:10:03.826631 | controller | - ' fi' 2026-03-09 19:10:03.826636 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.826642 | controller | - ' ' 2026-03-09 19:10:03.826649 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.826653 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.826657 | controller | | base64 -d)' 2026-03-09 19:10:03.826662 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.826666 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.826670 | controller | - ' exit 1' 2026-03-09 19:10:03.826674 | controller | - ' fi' 2026-03-09 19:10:03.826679 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.826683 | controller | - ' ' 2026-03-09 19:10:03.826687 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.826691 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.826696 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.826700 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.826704 | controller | --timeout=300s' 2026-03-09 19:10:03.826709 | controller | - ' ' 2026-03-09 19:10:03.826713 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.826717 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.826721 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.826726 | controller | - ' ' 2026-03-09 19:10:03.826730 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826734 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826738 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.826743 | controller | - ' ' 2026-03-09 19:10:03.826747 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.826751 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.826755 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.826760 | controller | available"' 2026-03-09 19:10:03.826764 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.826768 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.826772 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.826777 | controller | - ' exit 1' 2026-03-09 19:10:03.826781 | controller | - ' fi' 2026-03-09 19:10:03.826785 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.826789 | controller | - ' ' 2026-03-09 19:10:03.826794 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.826807 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.826811 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826816 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.826820 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826824 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.826828 | controller | -- \' 2026-03-09 19:10:03.826833 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.826837 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.826841 | controller | - ' logger.go:42: 19:00:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.826846 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.826852 | controller | - ' logger.go:42: 19:00:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.826856 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.826860 | controller | found' 2026-03-09 19:10:03.826864 | controller | - ' logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.826869 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.826873 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.826877 | controller | - ' ' 2026-03-09 19:10:03.826882 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.826886 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.826890 | controller | - ' ' 2026-03-09 19:10:03.826894 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.826899 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.826903 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.826907 | controller | - ' exit 1' 2026-03-09 19:10:03.826914 | controller | - ' fi' 2026-03-09 19:10:03.826918 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.826922 | controller | - ' ' 2026-03-09 19:10:03.826926 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.826931 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.826935 | controller | | base64 -d)' 2026-03-09 19:10:03.826939 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.826943 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.826948 | controller | - ' exit 1' 2026-03-09 19:10:03.826952 | controller | - ' fi' 2026-03-09 19:10:03.826956 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.826961 | controller | - ' ' 2026-03-09 19:10:03.826965 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.826969 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.826973 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.826977 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.826983 | controller | --timeout=300s' 2026-03-09 19:10:03.826989 | controller | - ' ' 2026-03-09 19:10:03.826995 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.827001 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.827011 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.827017 | controller | - ' ' 2026-03-09 19:10:03.827021 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827025 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827031 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827036 | controller | - ' ' 2026-03-09 19:10:03.827040 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.827044 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.827049 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.827053 | controller | available"' 2026-03-09 19:10:03.827057 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.827061 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.827066 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.827070 | controller | - ' exit 1' 2026-03-09 19:10:03.827074 | controller | - ' fi' 2026-03-09 19:10:03.827078 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.827083 | controller | - ' ' 2026-03-09 19:10:03.827087 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.827099 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.827104 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827108 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.827112 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827117 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.827121 | controller | -- \' 2026-03-09 19:10:03.827125 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827130 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.827134 | controller | - ' logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.827138 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.827142 | controller | - ' logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.827160 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.827165 | controller | found' 2026-03-09 19:10:03.827169 | controller | - ' logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.827174 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.827178 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.827182 | controller | - ' ' 2026-03-09 19:10:03.827187 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.827191 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.827195 | controller | - ' ' 2026-03-09 19:10:03.827199 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.827204 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.827208 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.827214 | controller | - ' exit 1' 2026-03-09 19:10:03.827219 | controller | - ' fi' 2026-03-09 19:10:03.827223 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.827227 | controller | - ' ' 2026-03-09 19:10:03.827231 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.827236 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.827240 | controller | | base64 -d)' 2026-03-09 19:10:03.827244 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.827248 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.827253 | controller | - ' exit 1' 2026-03-09 19:10:03.827257 | controller | - ' fi' 2026-03-09 19:10:03.827261 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.827265 | controller | - ' ' 2026-03-09 19:10:03.827270 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.827274 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.827278 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.827282 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.827287 | controller | --timeout=300s' 2026-03-09 19:10:03.827291 | controller | - ' ' 2026-03-09 19:10:03.827295 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.827299 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.827304 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.827308 | controller | - ' ' 2026-03-09 19:10:03.827312 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827317 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827321 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827325 | controller | - ' ' 2026-03-09 19:10:03.827329 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.827334 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.827338 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.827343 | controller | available"' 2026-03-09 19:10:03.827349 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.827353 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.827358 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.827362 | controller | - ' exit 1' 2026-03-09 19:10:03.827366 | controller | - ' fi' 2026-03-09 19:10:03.827370 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.827375 | controller | - ' ' 2026-03-09 19:10:03.827379 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.827383 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.827395 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827400 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.827404 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827410 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.827415 | controller | -- \' 2026-03-09 19:10:03.827419 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827423 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.827427 | controller | - ' logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.827432 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.827436 | controller | - ' logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.827442 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.827446 | controller | found' 2026-03-09 19:10:03.827451 | controller | - ' logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-09 19:10:03.827455 | controller | [sh -c set -euo pipefail' 2026-03-09 19:10:03.827459 | controller | - ' NS="${NAMESPACE}"' 2026-03-09 19:10:03.827464 | controller | - ' ' 2026-03-09 19:10:03.827468 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-09 19:10:03.827472 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-09 19:10:03.827476 | controller | - ' ' 2026-03-09 19:10:03.827480 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-09 19:10:03.827485 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-09 19:10:03.827489 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-09 19:10:03.827493 | controller | - ' exit 1' 2026-03-09 19:10:03.827498 | controller | - ' fi' 2026-03-09 19:10:03.827502 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-09 19:10:03.827506 | controller | - ' ' 2026-03-09 19:10:03.827510 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-09 19:10:03.827515 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-09 19:10:03.827519 | controller | | base64 -d)' 2026-03-09 19:10:03.827523 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-09 19:10:03.827527 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-09 19:10:03.827532 | controller | - ' exit 1' 2026-03-09 19:10:03.827536 | controller | - ' fi' 2026-03-09 19:10:03.827540 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-09 19:10:03.827544 | controller | - ' ' 2026-03-09 19:10:03.827549 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-09 19:10:03.827555 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-09 19:10:03.827560 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-09 19:10:03.827564 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-09 19:10:03.827569 | controller | --timeout=300s' 2026-03-09 19:10:03.827573 | controller | - ' ' 2026-03-09 19:10:03.827577 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-09 19:10:03.827581 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-09 19:10:03.827586 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-09 19:10:03.827591 | controller | - ' ' 2026-03-09 19:10:03.827596 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827601 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827623 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-09 19:10:03.827631 | controller | - ' ' 2026-03-09 19:10:03.827635 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-09 19:10:03.827640 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-09 19:10:03.827644 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-09 19:10:03.827648 | controller | available"' 2026-03-09 19:10:03.827652 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-09 19:10:03.827657 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-09 19:10:03.827661 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-09 19:10:03.827666 | controller | - ' exit 1' 2026-03-09 19:10:03.827671 | controller | - ' fi' 2026-03-09 19:10:03.827677 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-09 19:10:03.827682 | controller | - ' ' 2026-03-09 19:10:03.827688 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-09 19:10:03.827694 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-09 19:10:03.827709 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827715 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-09 19:10:03.827721 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827726 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-09 19:10:03.827732 | controller | -- \' 2026-03-09 19:10:03.827738 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-09 19:10:03.827743 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-09 19:10:03.827751 | controller | - ' logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-09 19:10:03.827757 | controller | ac-watcher to be Ready...' 2026-03-09 19:10:03.827763 | controller | - ' logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-09 19:10:03.827767 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-09 19:10:03.827771 | controller | found' 2026-03-09 19:10:03.827776 | controller | - ' logger.go:42: 19:00:59 | appcred-tests/2-deploy-appcred | test step failed 2026-03-09 19:10:03.827780 | controller | 2-deploy-appcred' 2026-03-09 19:10:03.827784 | controller | - ' case.go:396: failed in step 2-deploy-appcred' 2026-03-09 19:10:03.827789 | controller | - ' case.go:398: command "NS=\"${NAMESPACE}\"\\n echo \"Waiting for KeystoneApplicationCredential..." 2026-03-09 19:10:03.827793 | controller | failed, exit status 1' 2026-03-09 19:10:03.827797 | controller | - ' logger.go:42: 19:00:59 | appcred-tests | skipping kubernetes event logging' 2026-03-09 19:10:03.827802 | controller | - === CONT kuttl/harness/watcher-notification 2026-03-09 19:10:03.827806 | controller | - ' logger.go:42: 19:00:59 | watcher-notification | Skipping creation of user-supplied 2026-03-09 19:10:03.827810 | controller | namespace: watcher-kuttl-default' 2026-03-09 19:10:03.827814 | controller | - ' logger.go:42: 19:00:59 | watcher-notification/0-cleanup-watcher | starting 2026-03-09 19:10:03.827819 | controller | test step 0-cleanup-watcher' 2026-03-09 19:10:03.827823 | controller | - ' logger.go:42: 19:00:59 | watcher-notification/0-cleanup-watcher | test step 2026-03-09 19:10:03.827827 | controller | completed 0-cleanup-watcher' 2026-03-09 19:10:03.827831 | controller | - ' logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827836 | controller | starting test step 1-deploy-with-notification' 2026-03-09 19:10:03.827840 | controller | - ' logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827848 | controller | Watcher:watcher-kuttl-default/watcher-kuttl created' 2026-03-09 19:10:03.827852 | controller | - ' logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827857 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.827861 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.827865 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.827870 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.827874 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.827878 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.827882 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.827887 | controller | == 1 ]' 2026-03-09 19:10:03.827891 | controller | - ' else' 2026-03-09 19:10:03.827895 | controller | - ' exit 1' 2026-03-09 19:10:03.827900 | controller | - ' fi' 2026-03-09 19:10:03.827904 | controller | - ' ]' 2026-03-09 19:10:03.827908 | controller | - ' logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827913 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.827917 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827921 | controller | Now using project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.827926 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827930 | controller | ++ head -1' 2026-03-09 19:10:03.827934 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827938 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.827943 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827947 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.827953 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827957 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.827961 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827966 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.827970 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827974 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.827978 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827982 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.827994 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.827999 | controller | error: Internal error occurred: error executing command in container: container 2026-03-09 19:10:03.828004 | controller | is not created or running' 2026-03-09 19:10:03.828008 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828012 | controller | ++ echo' 2026-03-09 19:10:03.828017 | controller | - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828021 | controller | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.828026 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828030 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.828034 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.828038 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.828045 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.828049 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.828053 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.828257 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.828267 | controller | == 1 ]' 2026-03-09 19:10:03.828272 | controller | - ' else' 2026-03-09 19:10:03.828276 | controller | - ' exit 1' 2026-03-09 19:10:03.828280 | controller | - ' fi' 2026-03-09 19:10:03.828285 | controller | - ' ]' 2026-03-09 19:10:03.828289 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828293 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.828301 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828305 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.828309 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828314 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.828318 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828323 | controller | ++ head -1' 2026-03-09 19:10:03.828329 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828334 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.828341 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828346 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.828350 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828355 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.828359 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828363 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.828368 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828372 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.828376 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828381 | controller | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.828385 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828389 | controller | ++ echo' 2026-03-09 19:10:03.828394 | controller | - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828398 | controller | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.828402 | controller | - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828407 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.828411 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.828415 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.828420 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.828424 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.828428 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.828433 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.828437 | controller | == 1 ]' 2026-03-09 19:10:03.828441 | controller | - ' else' 2026-03-09 19:10:03.828446 | controller | - ' exit 1' 2026-03-09 19:10:03.828450 | controller | - ' fi' 2026-03-09 19:10:03.828456 | controller | - ' ]' 2026-03-09 19:10:03.828461 | controller | - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828466 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.828471 | controller | - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828477 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.828483 | controller | - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828488 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.828494 | controller | - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828500 | controller | ++ head -1' 2026-03-09 19:10:03.828518 | controller | - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828524 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.828534 | controller | - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828540 | controller | + APIPOD=' 2026-03-09 19:10:03.828545 | controller | - ' logger.go:42: 19:01:04 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828551 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.828557 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.828562 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.828568 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.828573 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.828578 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.828584 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.828590 | controller | == 1 ]' 2026-03-09 19:10:03.828595 | controller | - ' else' 2026-03-09 19:10:03.828600 | controller | - ' exit 1' 2026-03-09 19:10:03.828606 | controller | - ' fi' 2026-03-09 19:10:03.828612 | controller | - ' ]' 2026-03-09 19:10:03.828617 | controller | - ' logger.go:42: 19:01:04 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828622 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.828626 | controller | - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828630 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.828634 | controller | - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828639 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.828643 | controller | - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828647 | controller | ++ head -1' 2026-03-09 19:10:03.828652 | controller | - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828656 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.828660 | controller | - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828664 | controller | + APIPOD=' 2026-03-09 19:10:03.828669 | controller | - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828673 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.828677 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.828681 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.828686 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.828690 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.828694 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.828701 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.828705 | controller | == 1 ]' 2026-03-09 19:10:03.828710 | controller | - ' else' 2026-03-09 19:10:03.828714 | controller | - ' exit 1' 2026-03-09 19:10:03.828718 | controller | - ' fi' 2026-03-09 19:10:03.828723 | controller | - ' ]' 2026-03-09 19:10:03.828727 | controller | - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828731 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.828735 | controller | - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828740 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.828744 | controller | - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828748 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.828752 | controller | - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828757 | controller | ++ head -1' 2026-03-09 19:10:03.828761 | controller | - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828765 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.828770 | controller | - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828774 | controller | + APIPOD=' 2026-03-09 19:10:03.828778 | controller | - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828782 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.828787 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.828791 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.828795 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.828800 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.828804 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.828808 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.828812 | controller | == 1 ]' 2026-03-09 19:10:03.828817 | controller | - ' else' 2026-03-09 19:10:03.828821 | controller | - ' exit 1' 2026-03-09 19:10:03.828825 | controller | - ' fi' 2026-03-09 19:10:03.828830 | controller | - ' ]' 2026-03-09 19:10:03.828834 | controller | - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828838 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.828842 | controller | - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828855 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.828860 | controller | - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828864 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.828868 | controller | - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828873 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.828877 | controller | - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828881 | controller | ++ head -1' 2026-03-09 19:10:03.828885 | controller | - ' logger.go:42: 19:01:08 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828890 | controller | + APIPOD=' 2026-03-09 19:10:03.828894 | controller | - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828898 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.828902 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.828907 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.828913 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.828917 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.828922 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.828926 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.828930 | controller | == 1 ]' 2026-03-09 19:10:03.828934 | controller | - ' else' 2026-03-09 19:10:03.828939 | controller | - ' exit 1' 2026-03-09 19:10:03.828943 | controller | - ' fi' 2026-03-09 19:10:03.828947 | controller | - ' ]' 2026-03-09 19:10:03.828952 | controller | - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828956 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.828963 | controller | - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828968 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.828972 | controller | - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828976 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.828981 | controller | - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828985 | controller | ++ head -1' 2026-03-09 19:10:03.828989 | controller | - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.828993 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.828998 | controller | - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829002 | controller | + APIPOD=' 2026-03-09 19:10:03.829007 | controller | - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829013 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.829019 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.829025 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.829031 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.829036 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.829042 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.829050 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.829056 | controller | == 1 ]' 2026-03-09 19:10:03.829063 | controller | - ' else' 2026-03-09 19:10:03.829068 | controller | - ' exit 1' 2026-03-09 19:10:03.829075 | controller | - ' fi' 2026-03-09 19:10:03.829080 | controller | - ' ]' 2026-03-09 19:10:03.829086 | controller | - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829092 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.829099 | controller | - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829104 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.829111 | controller | - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829118 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.829123 | controller | - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829130 | controller | ++ head -1' 2026-03-09 19:10:03.829137 | controller | - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829142 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.829197 | controller | - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829203 | controller | + APIPOD=' 2026-03-09 19:10:03.829210 | controller | - ' logger.go:42: 19:01:11 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829215 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.829219 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.829223 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.829228 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.829232 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.829236 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.829250 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.829255 | controller | == 1 ]' 2026-03-09 19:10:03.829259 | controller | - ' else' 2026-03-09 19:10:03.829264 | controller | - ' exit 1' 2026-03-09 19:10:03.829268 | controller | - ' fi' 2026-03-09 19:10:03.829272 | controller | - ' ]' 2026-03-09 19:10:03.829276 | controller | - ' logger.go:42: 19:01:11 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829281 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.829285 | controller | - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829289 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.829294 | controller | - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829298 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.829302 | controller | - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829306 | controller | ++ head -1' 2026-03-09 19:10:03.829311 | controller | - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829315 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.829319 | controller | - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829324 | controller | + APIPOD=' 2026-03-09 19:10:03.829328 | controller | - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829332 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.829336 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.829341 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.829345 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.829349 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.829354 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.829358 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.829362 | controller | == 1 ]' 2026-03-09 19:10:03.829366 | controller | - ' else' 2026-03-09 19:10:03.829371 | controller | - ' exit 1' 2026-03-09 19:10:03.829375 | controller | - ' fi' 2026-03-09 19:10:03.829379 | controller | - ' ]' 2026-03-09 19:10:03.829384 | controller | - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829388 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.829392 | controller | - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829396 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.829401 | controller | - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829405 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.829409 | controller | - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829413 | controller | ++ head -1' 2026-03-09 19:10:03.829422 | controller | - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829426 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.829430 | controller | - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829434 | controller | + APIPOD=' 2026-03-09 19:10:03.829439 | controller | - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829443 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.829447 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.829452 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.829456 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.829460 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.829464 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.829469 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.829473 | controller | == 1 ]' 2026-03-09 19:10:03.829477 | controller | - ' else' 2026-03-09 19:10:03.829482 | controller | - ' exit 1' 2026-03-09 19:10:03.829486 | controller | - ' fi' 2026-03-09 19:10:03.829490 | controller | - ' ]' 2026-03-09 19:10:03.829496 | controller | - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829501 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.829507 | controller | - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829511 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.829515 | controller | - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829520 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.829524 | controller | - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829528 | controller | ++ head -1' 2026-03-09 19:10:03.829532 | controller | - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829537 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.829543 | controller | - ' logger.go:42: 19:01:15 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829547 | controller | + APIPOD=' 2026-03-09 19:10:03.829551 | controller | - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829556 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.829567 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.829572 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.829576 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.829580 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.829585 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.829589 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.829593 | controller | == 1 ]' 2026-03-09 19:10:03.829597 | controller | - ' else' 2026-03-09 19:10:03.829602 | controller | - ' exit 1' 2026-03-09 19:10:03.829606 | controller | - ' fi' 2026-03-09 19:10:03.829610 | controller | - ' ]' 2026-03-09 19:10:03.829615 | controller | - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829619 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.829623 | controller | - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829628 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.829632 | controller | - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829639 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.829643 | controller | - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829647 | controller | ++ head -1' 2026-03-09 19:10:03.829652 | controller | - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829656 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.829660 | controller | - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829664 | controller | + APIPOD=' 2026-03-09 19:10:03.829669 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829673 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.829677 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.829682 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.829686 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.829690 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.829695 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.829699 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.829703 | controller | == 1 ]' 2026-03-09 19:10:03.829707 | controller | - ' else' 2026-03-09 19:10:03.829712 | controller | - ' exit 1' 2026-03-09 19:10:03.829716 | controller | - ' fi' 2026-03-09 19:10:03.829720 | controller | - ' ]' 2026-03-09 19:10:03.829725 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829729 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.829733 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829738 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.829742 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829746 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.829750 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829755 | controller | ++ head -1' 2026-03-09 19:10:03.829759 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829763 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.829768 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829772 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.829776 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829781 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.829785 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829789 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.829795 | controller | - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829800 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.829804 | controller | - ' logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829808 | controller | error: unable to upgrade connection: container not found ("watcher-api")' 2026-03-09 19:10:03.829813 | controller | - ' logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829817 | controller | ++ echo' 2026-03-09 19:10:03.829821 | controller | - ' logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829827 | controller | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.829831 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829836 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.829840 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.829844 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.829850 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.829854 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.829866 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.829871 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.829875 | controller | == 1 ]' 2026-03-09 19:10:03.829879 | controller | - ' else' 2026-03-09 19:10:03.829884 | controller | - ' exit 1' 2026-03-09 19:10:03.829888 | controller | - ' fi' 2026-03-09 19:10:03.829892 | controller | - ' ]' 2026-03-09 19:10:03.829896 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829901 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.829905 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829909 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.829914 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829918 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.829922 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829926 | controller | ++ head -1' 2026-03-09 19:10:03.829931 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829935 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.829939 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829944 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.829948 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829952 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.829973 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829978 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.829983 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829987 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.829991 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.829996 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.830001 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.830006 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.830012 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.830018 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.830025 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.830037 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.830043 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.830047 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.830051 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.830056 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.830060 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.830065 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.830069 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.830079 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.830084 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.830088 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.830092 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.830096 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.830101 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-09 19:10:03.830113 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.830118 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.830122 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.830126 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.830131 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.830135 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.830139 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.830144 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.830162 | controller | - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830166 | controller | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.830173 | controller | - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830178 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.830182 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.830186 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.830191 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.830195 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.830199 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.830204 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.830208 | controller | == 1 ]' 2026-03-09 19:10:03.830212 | controller | - ' else' 2026-03-09 19:10:03.830216 | controller | - ' exit 1' 2026-03-09 19:10:03.830221 | controller | - ' fi' 2026-03-09 19:10:03.830227 | controller | - ' ]' 2026-03-09 19:10:03.830232 | controller | - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830236 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.830240 | controller | - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830245 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.830249 | controller | - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830253 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.830257 | controller | - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830262 | controller | ++ head -1' 2026-03-09 19:10:03.830266 | controller | - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830270 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.830275 | controller | - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830279 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.830283 | controller | - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830287 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.830292 | controller | - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830296 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.830300 | controller | - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830304 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.830309 | controller | - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830313 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.830317 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.830322 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.830326 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.830332 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.830336 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.830340 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.830345 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.830349 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.830361 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.830365 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.830370 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.830374 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.830378 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.830384 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.830388 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.830393 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.830397 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.830401 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.830406 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-09 19:10:03.830410 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.830414 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.830418 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.830423 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.830427 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.830431 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.830435 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.830440 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.830444 | controller | - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830448 | controller | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.830453 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830457 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.830461 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.830465 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.830470 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.830474 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.830478 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.830483 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.830487 | controller | == 1 ]' 2026-03-09 19:10:03.830491 | controller | - ' else' 2026-03-09 19:10:03.830495 | controller | - ' exit 1' 2026-03-09 19:10:03.830500 | controller | - ' fi' 2026-03-09 19:10:03.830504 | controller | - ' ]' 2026-03-09 19:10:03.830508 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830512 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.830517 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830521 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.830525 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830530 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.830534 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830538 | controller | ++ head -1' 2026-03-09 19:10:03.830542 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830547 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.830551 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830557 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.830561 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830565 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.830570 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830574 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.830578 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830583 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.830593 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830599 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.830604 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.830608 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.830613 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.830617 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.830621 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.830626 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.830630 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.830634 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.830638 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.830643 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.830647 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.830651 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.830657 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.830662 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.830666 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.830670 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.830674 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.830679 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.830683 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-09 19:10:03.830687 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.830692 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.830696 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.830703 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.830708 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.830712 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.830716 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.830720 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.830725 | controller | - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830729 | controller | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.830733 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830737 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.830742 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.830746 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.830750 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.830754 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.830759 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.830763 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.830767 | controller | == 1 ]' 2026-03-09 19:10:03.830772 | controller | - ' else' 2026-03-09 19:10:03.830776 | controller | - ' exit 1' 2026-03-09 19:10:03.830780 | controller | - ' fi' 2026-03-09 19:10:03.830784 | controller | - ' ]' 2026-03-09 19:10:03.830789 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830793 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.830804 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830808 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.830813 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830817 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.830821 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830825 | controller | ++ head -1' 2026-03-09 19:10:03.830830 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830834 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.830838 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830843 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.830847 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830851 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.830864 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830869 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.830873 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830877 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.830881 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.830886 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.830892 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.830896 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.830901 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.830905 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.830909 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.830913 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.830918 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.830922 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.830926 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.830930 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.830935 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.830939 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.830943 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.830948 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.830952 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.830956 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.830960 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.830965 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.830969 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-09 19:10:03.830973 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.830977 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.830982 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.830986 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.830992 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.830996 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.831008 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.831014 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.831020 | controller | - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831027 | controller | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.831033 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831038 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.831043 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.831049 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.831053 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.831058 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.831062 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.831066 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.831071 | controller | == 1 ]' 2026-03-09 19:10:03.831075 | controller | - ' else' 2026-03-09 19:10:03.831079 | controller | - ' exit 1' 2026-03-09 19:10:03.831084 | controller | - ' fi' 2026-03-09 19:10:03.831088 | controller | - ' ]' 2026-03-09 19:10:03.831092 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831097 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.831101 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831105 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.831109 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831114 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.831118 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831122 | controller | ++ head -1' 2026-03-09 19:10:03.831127 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831131 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.831135 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831139 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.831155 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831162 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.831166 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831170 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.831175 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831179 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.831183 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831187 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.831192 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.831196 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.831201 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.831205 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.831227 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.831232 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.831236 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.831244 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.831249 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.831253 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.831260 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.831264 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.831275 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.831280 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.831284 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.831288 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.831293 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.831297 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.831301 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-09 19:10:03.831305 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.831310 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.831314 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.831318 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.831322 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.831327 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.831331 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.831335 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.831340 | controller | - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831344 | controller | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.831348 | controller | - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831352 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.831357 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.831361 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.831365 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.831369 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.831376 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.831380 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.831384 | controller | == 1 ]' 2026-03-09 19:10:03.831388 | controller | - ' else' 2026-03-09 19:10:03.831393 | controller | - ' exit 1' 2026-03-09 19:10:03.831397 | controller | - ' fi' 2026-03-09 19:10:03.831401 | controller | - ' ]' 2026-03-09 19:10:03.831406 | controller | - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831410 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.831414 | controller | - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831420 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.831425 | controller | - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831429 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.831433 | controller | - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831437 | controller | ++ head -1' 2026-03-09 19:10:03.831442 | controller | - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831446 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.831450 | controller | - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831454 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.831459 | controller | - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831463 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.831469 | controller | - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831473 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.831478 | controller | - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831482 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.831486 | controller | - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831491 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.831495 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.831499 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.831510 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.831515 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.831519 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.831524 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.831528 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.831533 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.831537 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.831541 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.831545 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.831550 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.831554 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.831558 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.831562 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.831567 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.831573 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.831577 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.831582 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-09 19:10:03.831586 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.831592 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.831596 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.831622 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.831627 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.831631 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.831635 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.831640 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.831644 | controller | - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831648 | controller | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.831653 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831657 | controller | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.831661 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.831666 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.831670 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.831674 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.831679 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.831683 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-09 19:10:03.831687 | controller | == 1 ]' 2026-03-09 19:10:03.831692 | controller | - ' else' 2026-03-09 19:10:03.831696 | controller | - ' exit 1' 2026-03-09 19:10:03.831700 | controller | - ' fi' 2026-03-09 19:10:03.831705 | controller | - ' ]' 2026-03-09 19:10:03.831709 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831713 | controller | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.831718 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831722 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.831726 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831731 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.831735 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831739 | controller | ++ head -1' 2026-03-09 19:10:03.831751 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831756 | controller | ++ grep -v ''^$''' 2026-03-09 19:10:03.831762 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831767 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.831771 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831775 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.831781 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831786 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-09 19:10:03.831790 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831794 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.831798 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831803 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.831807 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.831811 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.831816 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.831820 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.831825 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_tPQqozd_sGGulezPOtJ:z0biWUNGWkBBYFtk6XznMLz0-K6jxbul@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.831829 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.831833 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.831838 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.831842 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.831846 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.831850 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.831855 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.831859 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.831863 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.831867 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.831872 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.831876 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.831882 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.831886 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-09 19:10:03.831890 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.831895 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.831899 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.831903 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.831908 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.831914 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.831918 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.831923 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.831927 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831933 | controller | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.831937 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | 2026-03-09 19:10:03.831942 | controller | test step completed 1-deploy-with-notification' 2026-03-09 19:10:03.831953 | controller | - ' logger.go:42: 19:01:29 | watcher-notification/2-cleanup-watcher | starting 2026-03-09 19:10:03.831958 | controller | test step 2-cleanup-watcher' 2026-03-09 19:10:03.831962 | controller | - ' logger.go:42: 19:01:35 | watcher-notification/2-cleanup-watcher | test step 2026-03-09 19:10:03.831967 | controller | completed 2-cleanup-watcher' 2026-03-09 19:10:03.831971 | controller | - ' logger.go:42: 19:01:35 | watcher-notification | skipping kubernetes event logging' 2026-03-09 19:10:03.831975 | controller | - === CONT kuttl/harness/watcher-topology 2026-03-09 19:10:03.831979 | controller | - ' logger.go:42: 19:01:35 | watcher-topology | Skipping creation of user-supplied 2026-03-09 19:10:03.831984 | controller | namespace: watcher-kuttl-default' 2026-03-09 19:10:03.831988 | controller | - ' logger.go:42: 19:01:35 | watcher-topology/0-cleanup-watcher | starting test 2026-03-09 19:10:03.831992 | controller | step 0-cleanup-watcher' 2026-03-09 19:10:03.831997 | controller | - ' logger.go:42: 19:01:35 | watcher-topology/0-cleanup-watcher | test step completed 2026-03-09 19:10:03.832001 | controller | 0-cleanup-watcher' 2026-03-09 19:10:03.832006 | controller | - ' logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | starting 2026-03-09 19:10:03.832012 | controller | test step 1-deploy-with-topology' 2026-03-09 19:10:03.832018 | controller | - ' logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | Topology:watcher-kuttl-default/watcher-api 2026-03-09 19:10:03.832024 | controller | created' 2026-03-09 19:10:03.832030 | controller | - ' logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.832036 | controller | created' 2026-03-09 19:10:03.832042 | controller | - ' logger.go:42: 19:01:59 | watcher-topology/1-deploy-with-topology | test step 2026-03-09 19:10:03.832047 | controller | completed 1-deploy-with-topology' 2026-03-09 19:10:03.832051 | controller | - ' logger.go:42: 19:01:59 | watcher-topology/2-cleanup-watcher | starting test 2026-03-09 19:10:03.832055 | controller | step 2-cleanup-watcher' 2026-03-09 19:10:03.832060 | controller | - ' logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | running command: 2026-03-09 19:10:03.832064 | controller | [sh -c set -ex' 2026-03-09 19:10:03.832068 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.832073 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.832077 | controller | - ' ]' 2026-03-09 19:10:03.832081 | controller | - ' logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | + oc exec -n 2026-03-09 19:10:03.832086 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.832090 | controller | -c Type' 2026-03-09 19:10:03.832094 | controller | - ' logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.832099 | controller | - ' logger.go:42: 19:02:02 | watcher-topology/2-cleanup-watcher | + ''['' 0 == 2026-03-09 19:10:03.832103 | controller | 0 '']''' 2026-03-09 19:10:03.832107 | controller | - ' logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | running command: 2026-03-09 19:10:03.832111 | controller | [sh -c set -ex' 2026-03-09 19:10:03.832116 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.832122 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.832127 | controller | - ' ]' 2026-03-09 19:10:03.832131 | controller | - ' logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | + oc exec -n 2026-03-09 19:10:03.832135 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.832140 | controller | -c Type' 2026-03-09 19:10:03.832144 | controller | - ' logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.832169 | controller | - ' logger.go:42: 19:02:06 | watcher-topology/2-cleanup-watcher | + ''['' 0 == 2026-03-09 19:10:03.832174 | controller | 0 '']''' 2026-03-09 19:10:03.832178 | controller | - ' logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | running command: 2026-03-09 19:10:03.832182 | controller | [sh -c set -ex' 2026-03-09 19:10:03.832187 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.832191 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.832195 | controller | - ' ]' 2026-03-09 19:10:03.832199 | controller | - ' logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | + oc exec -n 2026-03-09 19:10:03.832204 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.832208 | controller | -c Type' 2026-03-09 19:10:03.832215 | controller | - ' logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.832219 | controller | - ' logger.go:42: 19:02:09 | watcher-topology/2-cleanup-watcher | + ''['' 0 == 2026-03-09 19:10:03.832223 | controller | 0 '']''' 2026-03-09 19:10:03.832227 | controller | - ' logger.go:42: 19:02:09 | watcher-topology/2-cleanup-watcher | test step completed 2026-03-09 19:10:03.832232 | controller | 2-cleanup-watcher' 2026-03-09 19:10:03.832236 | controller | - ' logger.go:42: 19:02:09 | watcher-topology | skipping kubernetes event logging' 2026-03-09 19:10:03.832240 | controller | - === CONT kuttl/harness/watcher-tls-certs-change 2026-03-09 19:10:03.832244 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change | Skipping creation of user-supplied 2026-03-09 19:10:03.832249 | controller | namespace: watcher-kuttl-default' 2026-03-09 19:10:03.832253 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/0-cleanup-watcher | starting 2026-03-09 19:10:03.832257 | controller | test step 0-cleanup-watcher' 2026-03-09 19:10:03.832262 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/0-cleanup-watcher | test 2026-03-09 19:10:03.832266 | controller | step completed 0-cleanup-watcher' 2026-03-09 19:10:03.832279 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | starting 2026-03-09 19:10:03.832284 | controller | test step 1-deploy-with-tlse' 2026-03-09 19:10:03.832288 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc 2026-03-09 19:10:03.832293 | controller | created' 2026-03-09 19:10:03.832297 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc 2026-03-09 19:10:03.832301 | controller | created' 2026-03-09 19:10:03.832305 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.832310 | controller | created' 2026-03-09 19:10:03.832316 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.832320 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.832324 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832329 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.832333 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.832337 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832341 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832349 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.832353 | controller | - ' exit 1' 2026-03-09 19:10:03.832357 | controller | - ' fi' 2026-03-09 19:10:03.832362 | controller | - ' ' 2026-03-09 19:10:03.832366 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832370 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.832374 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.832379 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832383 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832387 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.832391 | controller | - ' exit 1' 2026-03-09 19:10:03.832396 | controller | - ' fi' 2026-03-09 19:10:03.832400 | controller | - ' ]' 2026-03-09 19:10:03.832404 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.832408 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.832413 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.832417 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.832421 | controller | - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.832425 | controller | - ' logger.go:42: 19:02:10 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.832430 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.832436 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832440 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.832444 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.832448 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832453 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832457 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.832461 | controller | - ' exit 1' 2026-03-09 19:10:03.832465 | controller | - ' fi' 2026-03-09 19:10:03.832470 | controller | - ' ' 2026-03-09 19:10:03.832474 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832478 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.832482 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.832487 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832491 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832495 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.832499 | controller | - ' exit 1' 2026-03-09 19:10:03.832504 | controller | - ' fi' 2026-03-09 19:10:03.832508 | controller | - ' ]' 2026-03-09 19:10:03.832512 | controller | - ' logger.go:42: 19:02:10 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.832516 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.832521 | controller | - ' logger.go:42: 19:02:11 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.832525 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.832529 | controller | - ' logger.go:42: 19:02:11 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.832533 | controller | - ' logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.832538 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.832543 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832548 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.832552 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.832556 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832568 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832572 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.832578 | controller | - ' exit 1' 2026-03-09 19:10:03.832584 | controller | - ' fi' 2026-03-09 19:10:03.832590 | controller | - ' ' 2026-03-09 19:10:03.832596 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832601 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.832607 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.832613 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832619 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832624 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.832630 | controller | - ' exit 1' 2026-03-09 19:10:03.832636 | controller | - ' fi' 2026-03-09 19:10:03.832642 | controller | - ' ]' 2026-03-09 19:10:03.832647 | controller | - ' logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.832652 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.832656 | controller | - ' logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.832660 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.832664 | controller | - ' logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.832669 | controller | - ' logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.832673 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.832677 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832681 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.832686 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.832690 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832694 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832698 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.832703 | controller | - ' exit 1' 2026-03-09 19:10:03.832707 | controller | - ' fi' 2026-03-09 19:10:03.832711 | controller | - ' ' 2026-03-09 19:10:03.832715 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832719 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.832724 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.832728 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832732 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832736 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.832740 | controller | - ' exit 1' 2026-03-09 19:10:03.832745 | controller | - ' fi' 2026-03-09 19:10:03.832749 | controller | - ' ]' 2026-03-09 19:10:03.832753 | controller | - ' logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.832757 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.832765 | controller | - ' logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.832771 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.832776 | controller | - ' logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.832780 | controller | - ' logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.832784 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.832788 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832793 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.832797 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.832801 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832805 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832810 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.832814 | controller | - ' exit 1' 2026-03-09 19:10:03.832818 | controller | - ' fi' 2026-03-09 19:10:03.832823 | controller | - ' ' 2026-03-09 19:10:03.832827 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832831 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.832835 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.832840 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832844 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832848 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.832852 | controller | - ' exit 1' 2026-03-09 19:10:03.832857 | controller | - ' fi' 2026-03-09 19:10:03.832861 | controller | - ' ]' 2026-03-09 19:10:03.832865 | controller | - ' logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.832869 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.832874 | controller | - ' logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.832886 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.832891 | controller | - ' logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.832895 | controller | - ' logger.go:42: 19:02:15 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.832900 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.832904 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832908 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.832913 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.832917 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832921 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832926 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.832930 | controller | - ' exit 1' 2026-03-09 19:10:03.832934 | controller | - ' fi' 2026-03-09 19:10:03.832938 | controller | - ' ' 2026-03-09 19:10:03.832943 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.832947 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.832951 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.832955 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.832960 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.832964 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.832970 | controller | - ' exit 1' 2026-03-09 19:10:03.832974 | controller | - ' fi' 2026-03-09 19:10:03.832978 | controller | - ' ]' 2026-03-09 19:10:03.832983 | controller | - ' logger.go:42: 19:02:15 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.832987 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.832991 | controller | - ' logger.go:42: 19:02:16 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.832995 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.833000 | controller | - ' logger.go:42: 19:02:16 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.833004 | controller | - ' logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.833010 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.833016 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833022 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.833028 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.833034 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833039 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833043 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.833047 | controller | - ' exit 1' 2026-03-09 19:10:03.833052 | controller | - ' fi' 2026-03-09 19:10:03.833056 | controller | - ' ' 2026-03-09 19:10:03.833060 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833064 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.833069 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.833073 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833077 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833081 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.833085 | controller | - ' exit 1' 2026-03-09 19:10:03.833090 | controller | - ' fi' 2026-03-09 19:10:03.833094 | controller | - ' ]' 2026-03-09 19:10:03.833099 | controller | - ' logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.833103 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.833110 | controller | - ' logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.833115 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.833120 | controller | - ' logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.833125 | controller | - ' logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.833130 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.833135 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833140 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.833162 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.833169 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833174 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833178 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.833182 | controller | - ' exit 1' 2026-03-09 19:10:03.833186 | controller | - ' fi' 2026-03-09 19:10:03.833191 | controller | - ' ' 2026-03-09 19:10:03.833195 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833201 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.833206 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.833219 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833223 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833228 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.833232 | controller | - ' exit 1' 2026-03-09 19:10:03.833236 | controller | - ' fi' 2026-03-09 19:10:03.833241 | controller | - ' ]' 2026-03-09 19:10:03.833247 | controller | - ' logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.833252 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.833256 | controller | - ' logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.833260 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.833264 | controller | - ' logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.833269 | controller | - ' logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.833273 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.833277 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833281 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.833286 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.833290 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833294 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833298 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.833303 | controller | - ' exit 1' 2026-03-09 19:10:03.833307 | controller | - ' fi' 2026-03-09 19:10:03.833311 | controller | - ' ' 2026-03-09 19:10:03.833315 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833320 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.833324 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.833328 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833333 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833337 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.833341 | controller | - ' exit 1' 2026-03-09 19:10:03.833348 | controller | - ' fi' 2026-03-09 19:10:03.833352 | controller | - ' ]' 2026-03-09 19:10:03.833357 | controller | - ' logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.833361 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.833365 | controller | - ' logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.833369 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.833374 | controller | - ' logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.833378 | controller | - ' logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.833382 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.833387 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833391 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.833395 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.833399 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833406 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833410 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.833414 | controller | - ' exit 1' 2026-03-09 19:10:03.833418 | controller | - ' fi' 2026-03-09 19:10:03.833423 | controller | - ' ' 2026-03-09 19:10:03.833427 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833431 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.833435 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.833440 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833444 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833448 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.833453 | controller | - ' exit 1' 2026-03-09 19:10:03.833457 | controller | - ' fi' 2026-03-09 19:10:03.833461 | controller | - ' ]' 2026-03-09 19:10:03.833465 | controller | - ' logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.833470 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.833474 | controller | - ' logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-09 19:10:03.833478 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-09 19:10:03.833482 | controller | - ' logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.833487 | controller | - ' logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.833491 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.833498 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833509 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.833513 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.833517 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833522 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833526 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.833530 | controller | - ' exit 1' 2026-03-09 19:10:03.833535 | controller | - ' fi' 2026-03-09 19:10:03.833539 | controller | - ' ' 2026-03-09 19:10:03.833543 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833547 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.833551 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.833556 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833560 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833564 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.833568 | controller | - ' exit 1' 2026-03-09 19:10:03.833573 | controller | - ' fi' 2026-03-09 19:10:03.833577 | controller | - ' ]' 2026-03-09 19:10:03.833581 | controller | - ' logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.833585 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.833590 | controller | - ' logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | error: 2026-03-09 19:10:03.833594 | controller | unable to upgrade connection: container not found ("watcher-api")' 2026-03-09 19:10:03.833598 | controller | - ' logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.833602 | controller | - ' logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.833608 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.833613 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833617 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.833621 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.833626 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833630 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833634 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.833639 | controller | - ' exit 1' 2026-03-09 19:10:03.833643 | controller | - ' fi' 2026-03-09 19:10:03.833647 | controller | - ' ' 2026-03-09 19:10:03.833651 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833656 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.833662 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.833666 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833670 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833675 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.833684 | controller | - ' exit 1' 2026-03-09 19:10:03.833690 | controller | - ' fi' 2026-03-09 19:10:03.833696 | controller | - ' ]' 2026-03-09 19:10:03.833702 | controller | - ' logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.833707 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.833713 | controller | - ' logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | error: 2026-03-09 19:10:03.833719 | controller | unable to upgrade connection: container not found ("watcher-api")' 2026-03-09 19:10:03.833725 | controller | - ' logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-09 19:10:03.833730 | controller | - ' logger.go:42: 19:02:24 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.833736 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.833742 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833747 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.833752 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.833758 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833763 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833769 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.833774 | controller | - ' exit 1' 2026-03-09 19:10:03.833779 | controller | - ' fi' 2026-03-09 19:10:03.833785 | controller | - ' ' 2026-03-09 19:10:03.833790 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.833796 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.833801 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.833807 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.833813 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.833818 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.833823 | controller | - ' exit 1' 2026-03-09 19:10:03.833829 | controller | - ' fi' 2026-03-09 19:10:03.833835 | controller | - ' ]' 2026-03-09 19:10:03.833840 | controller | - ' logger.go:42: 19:02:24 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.833854 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.833861 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-09 19:10:03.833865 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.833870 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.833876 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.833881 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.833885 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.833890 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.833894 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.833898 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.833902 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.833907 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.833911 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.833915 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.833919 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.833924 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.833928 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.833932 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.833937 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.833941 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.833948 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.833952 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.833956 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.833961 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.833965 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.833969 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.833975 | controller | --decode' 2026-03-09 19:10:03.833980 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.833984 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.833988 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-09 19:10:03.833993 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.833997 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.834001 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.834014 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.834021 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.834027 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.834034 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.834040 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.834045 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.834050 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.834054 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.834058 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.834062 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.834067 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.834071 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.834075 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.834080 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.834084 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.834088 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.834093 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.834099 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.834103 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.834112 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.834116 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.834121 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.834125 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.834129 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.834134 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.834138 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.834142 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.834162 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.834167 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.834171 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.834183 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.834188 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.834192 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.834197 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.834201 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.834205 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.834209 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.834214 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.834218 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.834222 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.834227 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.834231 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.834235 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.834239 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.834244 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.834250 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.834257 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.834261 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.834266 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.834270 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.834274 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.834278 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.834283 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.834287 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.834291 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.834295 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.834300 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.834304 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.834308 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.834319 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.834324 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.834328 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.834332 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.834337 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.834341 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.834345 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.834350 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.834354 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-09 19:10:03.834358 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-09 19:10:03.834363 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.834367 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.834374 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.834379 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.834383 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.834389 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.834393 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.834398 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.834402 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.834406 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.834410 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.834415 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.834419 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.834423 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.834427 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.834432 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.834436 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.834440 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.834444 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.834449 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.834453 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.834464 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.834469 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.834473 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.834477 | controller | --decode' 2026-03-09 19:10:03.834482 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.834486 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.834492 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-09 19:10:03.834496 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.834501 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.834505 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.834509 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.834514 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.834518 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.834524 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.834528 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.834532 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.834537 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.834541 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.834545 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.834549 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.834554 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.834558 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.834562 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.834567 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.834571 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.834575 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.834579 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.834584 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.834588 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.834592 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.834597 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.834601 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.834607 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.834611 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.834623 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.834627 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.834632 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.834636 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.834644 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.834648 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.834653 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.834657 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.834661 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.834665 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.834670 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.834674 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.834678 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.834683 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.834687 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.834691 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.834695 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.834700 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.834704 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.834708 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.834713 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.834717 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.834723 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.834727 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.834731 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.834736 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.834740 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.834746 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.834756 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.834760 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.834765 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.834769 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.834773 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.834777 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.834782 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.834786 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.834790 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.834794 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.834799 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.834803 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.834807 | controller | - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.834812 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.834816 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.834820 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.834825 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.834829 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.834833 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.834837 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.834842 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.834846 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.834852 | controller | - ' exit 1' 2026-03-09 19:10:03.834856 | controller | - ' fi' 2026-03-09 19:10:03.834861 | controller | - ' ' 2026-03-09 19:10:03.834865 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.834869 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.834873 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.834878 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.834882 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.834886 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.834891 | controller | - ' exit 1' 2026-03-09 19:10:03.834895 | controller | - ' fi' 2026-03-09 19:10:03.834899 | controller | - ' ]' 2026-03-09 19:10:03.834903 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.834908 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.834912 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-09 19:10:03.834917 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.834922 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.834928 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.834932 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.834936 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.834941 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.834945 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.834955 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.834960 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.834966 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.834970 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.834974 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.834979 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.834983 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.834987 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.834992 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.834996 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.835002 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.835007 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.835013 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.835019 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.835025 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.835031 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.835037 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.835043 | controller | --decode' 2026-03-09 19:10:03.835048 | controller | - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.835054 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.835060 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-09 19:10:03.835065 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.835071 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.835076 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.835082 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.835088 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.835092 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.835096 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.835100 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.835105 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.835109 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.835113 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.835127 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.835132 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.835136 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.835140 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.835159 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.835165 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.835170 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.835174 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.835178 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.835183 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.835188 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.835192 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.835197 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.835201 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.835205 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.835210 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.835214 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.835218 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.835222 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.835227 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.835231 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.835235 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.835239 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.835244 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.835248 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.835252 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.835258 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.835263 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.835267 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.835273 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.835283 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.835288 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.835292 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.835297 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.835301 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.835305 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.835310 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.835314 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.835318 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.835323 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.835327 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.835331 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.835335 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.835340 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.835344 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.835348 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.835353 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.835357 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.835361 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.835366 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.835371 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.835376 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.835380 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.835384 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.835390 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.835394 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.835399 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.835403 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.835407 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.835412 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-09 19:10:03.835416 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-09 19:10:03.835425 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.835430 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.835434 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.835439 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.835443 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.835447 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.835451 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.835456 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.835460 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.835464 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.835468 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.835473 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.835477 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.835481 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.835486 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.835490 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.835494 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.835500 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.835513 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.835517 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.835522 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.835526 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.835530 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.835535 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.835539 | controller | --decode' 2026-03-09 19:10:03.835543 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.835547 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.835552 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-09 19:10:03.835556 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.835560 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.835565 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.835569 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.835579 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.835583 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.835587 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.835592 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.835616 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.835622 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.835627 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.835631 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.835635 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.835643 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.835647 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.835652 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.835656 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.835662 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.835668 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.835672 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.835676 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.835681 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.835685 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.835689 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.835694 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.835698 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.835702 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.835707 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.835711 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.835715 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.835719 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.835724 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.835728 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.835732 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.835742 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.835747 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.835751 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.835756 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.835760 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.835764 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.835768 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.835773 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.835779 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.835785 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.835790 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.835794 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.835798 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.835803 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.835807 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.835811 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.835815 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.835820 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.835824 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.835828 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.835832 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.835837 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.835841 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.835845 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.835850 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.835854 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.835858 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.835862 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.835867 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.835876 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.835881 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.835885 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.835893 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.835897 | controller | - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.835901 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.835906 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.835910 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.835914 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.835918 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.835923 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.835927 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.835931 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.835936 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.835940 | controller | - ' exit 1' 2026-03-09 19:10:03.835944 | controller | - ' fi' 2026-03-09 19:10:03.835948 | controller | - ' ' 2026-03-09 19:10:03.835953 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.835957 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.835961 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.835965 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.835970 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.835974 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.835978 | controller | - ' exit 1' 2026-03-09 19:10:03.835982 | controller | - ' fi' 2026-03-09 19:10:03.835987 | controller | - ' ]' 2026-03-09 19:10:03.835991 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.835995 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.835999 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-09 19:10:03.836004 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.836010 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.836016 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.836022 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.836028 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.836034 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.836040 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.836044 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.836048 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.836052 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.836059 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.836063 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.836067 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.836072 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.836082 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.836087 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.836091 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.836095 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.836099 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.836104 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.836108 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.836115 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.836119 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.836123 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.836128 | controller | --decode' 2026-03-09 19:10:03.836132 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.836136 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.836141 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-09 19:10:03.836164 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.836172 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.836178 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.836182 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.836187 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.836191 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.836195 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.836199 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.836204 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.836210 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.836214 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.836218 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.836223 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.836227 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.836231 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.836236 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.836240 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.836244 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.836254 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.836259 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.836263 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.836268 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.836272 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.836277 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.836281 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.836285 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.836291 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.836296 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.836301 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.836306 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.836310 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.836314 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.836319 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.836323 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.836329 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.836334 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.836338 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.836342 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.836347 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.836351 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.836355 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.836359 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.836364 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.836368 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.836372 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.836377 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.836381 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.836385 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.836389 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.836399 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.836404 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.836408 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.836415 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.836419 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.836424 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.836428 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.836432 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.836436 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.836444 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.836448 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.836452 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.836457 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.836461 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.836465 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.836469 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.836474 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.836478 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.836482 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.836487 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.836491 | controller | - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.836495 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-09 19:10:03.836500 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-09 19:10:03.836504 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.836508 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.836513 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.836517 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.836521 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.836526 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.836530 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.836534 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.836545 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.836550 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.836554 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.836559 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.836565 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.836569 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.836573 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.836577 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.836582 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.836586 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.836590 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.836594 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.836599 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.836603 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.836607 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.836612 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.836616 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.836620 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.836624 | controller | --decode' 2026-03-09 19:10:03.836629 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-09 19:10:03.836633 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.836637 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.836643 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.836647 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.836652 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.836656 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.836660 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.836664 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.836669 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.836674 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.836681 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.836685 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.836695 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.836699 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.836704 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.836708 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.836712 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.836717 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.836721 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.836725 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.836729 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.836734 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.836738 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.836742 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.836747 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.836751 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.836755 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.836760 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.836764 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.836768 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.836772 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.836777 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.836781 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.836785 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.836791 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.836797 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.836801 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.836805 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.836810 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.836814 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.836818 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.836822 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.836832 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.836837 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.836841 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.836846 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.836850 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.836854 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.836858 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.836863 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.836867 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.836871 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.836876 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.836880 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.836884 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.836888 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.836893 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.836899 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.836903 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.836909 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.836913 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.836917 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.836922 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.836926 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.836930 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.836934 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.836939 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.836943 | controller | - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.836947 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.836952 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.836956 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.836960 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.836965 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.836969 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.836973 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.836977 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.836987 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.836991 | controller | - ' exit 1' 2026-03-09 19:10:03.836996 | controller | - ' fi' 2026-03-09 19:10:03.837000 | controller | - ' ' 2026-03-09 19:10:03.837005 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.837011 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.837017 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.837023 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.837029 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.837035 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.837041 | controller | - ' exit 1' 2026-03-09 19:10:03.837045 | controller | - ' fi' 2026-03-09 19:10:03.837049 | controller | - ' ]' 2026-03-09 19:10:03.837053 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.837058 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.837062 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-09 19:10:03.837066 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.837071 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.837075 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.837081 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.837086 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.837090 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.837094 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.837099 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.837103 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.837107 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.837111 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.837116 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.837122 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.837127 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.837131 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.837135 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.837140 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.837157 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.837164 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.837168 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.837172 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.837177 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.837181 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.837186 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.837196 | controller | --decode' 2026-03-09 19:10:03.837201 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.837206 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.837210 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-09 19:10:03.837216 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.837222 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.837231 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.837237 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.837243 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.837249 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.837254 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.837260 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.837266 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.837271 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.837276 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.837282 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.837287 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.837293 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.837300 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.837304 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.837309 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.837313 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.837317 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.837321 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.837326 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.837330 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.837334 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.837339 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.837343 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.837347 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.837351 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.837357 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.837368 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.837372 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.837377 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.837381 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.837385 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.837390 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.837394 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.837398 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.837402 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.837407 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.837411 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.837415 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.837423 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.837427 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.837431 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.837436 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.837440 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.837444 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.837449 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.837453 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.837457 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.837461 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.837466 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.837472 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.837476 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.837480 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.837485 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.837489 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.837493 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.837502 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.837507 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.837511 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.837516 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.837520 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.837524 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.837529 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.837535 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.837539 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.837543 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.837548 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.837552 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.837556 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.837561 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-09 19:10:03.837565 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-09 19:10:03.837569 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.837573 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.837578 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.837582 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.837586 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.837592 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.837596 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.837601 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.837605 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.837609 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.837614 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.837618 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.837622 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.837626 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.837631 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.837635 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.837644 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.837649 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.837653 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.837658 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.837663 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.837668 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.837672 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.837676 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.837681 | controller | --decode' 2026-03-09 19:10:03.837685 | controller | - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.837689 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.837694 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-09 19:10:03.837698 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.837702 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.837706 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.837711 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.837717 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.837721 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.837725 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.837730 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.837734 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.837738 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.837742 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.837747 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.837751 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.837755 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.837760 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.837764 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.837768 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.837772 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.837777 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.837781 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.837790 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.837795 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.837799 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.837804 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.837808 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.837812 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.837818 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.837823 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.837827 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.837833 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.837837 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.837841 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.837846 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.837850 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.837854 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.837858 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.837863 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.837867 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.837871 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.837876 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.837880 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.837884 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.837888 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.837893 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.837897 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.837901 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.837906 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.837910 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.837916 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.837920 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.837924 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.837934 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.837939 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.837945 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.837949 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.837954 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.837958 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.837962 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.837966 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.837971 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.837975 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.837979 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.837983 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.837988 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.837992 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.837998 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.838004 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.838009 | controller | - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.838015 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.838021 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.838027 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.838033 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.838039 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.838044 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.838050 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.838055 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.838061 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.838066 | controller | - ' exit 1' 2026-03-09 19:10:03.838072 | controller | - ' fi' 2026-03-09 19:10:03.838077 | controller | - ' ' 2026-03-09 19:10:03.838083 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.838088 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.838093 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.838098 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.838108 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.838116 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.838122 | controller | - ' exit 1' 2026-03-09 19:10:03.838127 | controller | - ' fi' 2026-03-09 19:10:03.838133 | controller | - ' ]' 2026-03-09 19:10:03.838139 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.838159 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.838169 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-09 19:10:03.838175 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.838181 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.838201 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.838211 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.838217 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.838223 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.838228 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.838234 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.838241 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.838246 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.838250 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.838254 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.838259 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.838263 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.838267 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.838271 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.838276 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.838280 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.838284 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.838288 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.838295 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.838300 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.838304 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.838308 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.838312 | controller | --decode' 2026-03-09 19:10:03.838317 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.838321 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.838325 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-09 19:10:03.838330 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.838334 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.838338 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.838343 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.838347 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.838355 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.838367 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.838371 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.838376 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.838380 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.838384 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.838389 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.838393 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.838397 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.838401 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.838406 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.838410 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.838414 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.838419 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.838425 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.838429 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.838433 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.838438 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.838442 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.838446 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.838450 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.838455 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.838459 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.838463 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.838468 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.838472 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.838478 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.838482 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.838486 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.838491 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.838495 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.838506 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.838511 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.838515 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.838519 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.838524 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.838528 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.838532 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.838536 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.838543 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.838547 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.838552 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.838557 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.838563 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.838569 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.838575 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.838580 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.838586 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.838592 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.838598 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.838606 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.838612 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.838618 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.838623 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.838628 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.838632 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.838636 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.838640 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.838645 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.838649 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.838661 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.838666 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.838670 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.838677 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.838681 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.838686 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-09 19:10:03.838690 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-09 19:10:03.838694 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.838699 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.838703 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.838707 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.838712 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.838716 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.838720 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.838724 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.838729 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.838733 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.838737 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.838748 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.838752 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.838756 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.838761 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.838765 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.838769 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.838773 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.838778 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.838782 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.838786 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.838791 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.838797 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.838801 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.838805 | controller | --decode' 2026-03-09 19:10:03.838810 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.838814 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.838825 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-09 19:10:03.838830 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.838834 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.838839 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.838843 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.838847 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.838852 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.838856 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.838860 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.838865 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.838871 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.838877 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.838882 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.838890 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.838896 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.838902 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.838908 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.838913 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.838919 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.838926 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.838931 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.838937 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.838942 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.838946 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.838950 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.838955 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.838959 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.838963 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.838968 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.838972 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.838976 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.838989 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.838994 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.838998 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.839002 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.839007 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.839011 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.839015 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.839022 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.839026 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.839030 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.839035 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.839039 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.839043 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.839048 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.839052 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.839056 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.839062 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.839067 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.839071 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.839075 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.839079 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.839084 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.839088 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.839092 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.839096 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.839101 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.839105 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.839109 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.839119 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.839123 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.839130 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.839134 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.839138 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.839142 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.839162 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.839166 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.839170 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.839175 | controller | - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.839179 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.839183 | controller | - ' logger.go:42: 19:02:33 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.839188 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.839194 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.839198 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.839202 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.839207 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.839211 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.839215 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.839219 | controller | - ' exit 1' 2026-03-09 19:10:03.839224 | controller | - ' fi' 2026-03-09 19:10:03.839228 | controller | - ' ' 2026-03-09 19:10:03.839232 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.839237 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.839241 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.839245 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.839249 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.839254 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.839258 | controller | - ' exit 1' 2026-03-09 19:10:03.839262 | controller | - ' fi' 2026-03-09 19:10:03.839266 | controller | - ' ]' 2026-03-09 19:10:03.839271 | controller | - ' logger.go:42: 19:02:33 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.839275 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.839279 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-09 19:10:03.839283 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.839288 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.839292 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.839296 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.839301 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.839305 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.839309 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.839313 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.839318 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.839330 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.839336 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.839342 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.839348 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.839353 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.839362 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.839372 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.839378 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.839384 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.839390 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.839395 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.839399 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.839403 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.839408 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.839412 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.839416 | controller | --decode' 2026-03-09 19:10:03.839421 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.839425 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.839429 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-09 19:10:03.839434 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.839438 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.839442 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.839446 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.839451 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.839455 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.839459 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.839464 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.839468 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.839472 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.839476 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.839481 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.839487 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.839499 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.839504 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.839508 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.839513 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.839527 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.839531 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.839535 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.839540 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.839544 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.839548 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.839553 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.839559 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.839563 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.839567 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.839572 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.839576 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.839580 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.839584 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.839589 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.839615 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.839620 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.839624 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.839628 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.839633 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.839637 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.839643 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.839647 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.839652 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.839656 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.839660 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.839666 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.839678 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.839682 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.839687 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.839691 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.839697 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.839703 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.839709 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.839714 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.839720 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.839726 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.839732 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.839737 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.839742 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.839748 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.839753 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.839759 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.839786 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.839791 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.839800 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.839806 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.839811 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.839817 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.839822 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.839828 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.839834 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.839840 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.839846 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-09 19:10:03.839851 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-09 19:10:03.839861 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.839867 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.839874 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.839891 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.839898 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.839903 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.839907 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.839911 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.839916 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.839920 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.839924 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.839928 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.839933 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.839937 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.839941 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.839948 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.839952 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.839956 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.839961 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.839965 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.839969 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.839974 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.839978 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.839982 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.839987 | controller | --decode' 2026-03-09 19:10:03.839991 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.839995 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.840000 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-09 19:10:03.840004 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.840010 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.840016 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.840025 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.840031 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.840037 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.840043 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.840058 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.840064 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.840070 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.840075 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.840081 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.840086 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.840092 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.840100 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.840105 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.840109 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.840113 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.840118 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.840122 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.840126 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.840131 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.840135 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.840139 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.840162 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.840170 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.840175 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.840179 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.840185 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.840190 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.840194 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.840198 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.840203 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.840207 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.840211 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.840216 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.840220 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.840231 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.840235 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.840242 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.840247 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.840251 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.840255 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.840260 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.840264 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.840268 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.840272 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.840277 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.840281 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.840285 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.840290 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.840296 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.840300 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.840304 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.840308 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.840313 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.840317 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.840321 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.840326 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.840330 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.840334 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.840338 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.840343 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.840350 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.840355 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.840359 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.840369 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.840373 | controller | - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.840378 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.840382 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-09 19:10:03.840386 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.840391 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.840395 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.840399 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-09 19:10:03.840404 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.840408 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.840412 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-09 19:10:03.840417 | controller | - ' exit 1' 2026-03-09 19:10:03.840421 | controller | - ' fi' 2026-03-09 19:10:03.840425 | controller | - ' ' 2026-03-09 19:10:03.840429 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-09 19:10:03.840434 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.840438 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-09 19:10:03.840442 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.840446 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.840451 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-09 19:10:03.840455 | controller | - ' exit 1' 2026-03-09 19:10:03.840459 | controller | - ' fi' 2026-03-09 19:10:03.840463 | controller | - ' ]' 2026-03-09 19:10:03.840468 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.840472 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.840476 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-09 19:10:03.840480 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.840485 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.840489 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.840493 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.840498 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.840502 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.840508 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.840515 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.840519 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.840523 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.840527 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.840532 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.840536 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.840540 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.840545 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.840549 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.840553 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.840564 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.840569 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.840573 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.840578 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.840582 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.840586 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.840591 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.840595 | controller | --decode' 2026-03-09 19:10:03.840599 | controller | - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.840603 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.840608 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-09 19:10:03.840612 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.840616 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.840622 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.840627 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.840631 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.840636 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.840642 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.840646 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.840652 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.840656 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.840660 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.840665 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.840669 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.840673 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.840678 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.840682 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.840686 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.840690 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.840695 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.840699 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.840703 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.840708 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.840718 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.840723 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.840728 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.840732 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.840736 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.840741 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.840745 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.840749 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.840754 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.840760 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.840764 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.840768 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.840774 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.840778 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.840783 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.840788 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.840793 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.840797 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.840801 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.840806 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.840810 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.840814 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.840818 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.840823 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.840827 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.840831 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.840836 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.840840 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.840844 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.840849 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.840858 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.840863 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.840867 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.840873 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.840878 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.840882 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.840888 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.840892 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.840897 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.840901 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.840905 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.840909 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.840914 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.840918 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.840922 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.840927 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.840931 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.840935 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.840940 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-09 19:10:03.840944 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-09 19:10:03.840948 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.840953 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.840957 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.840961 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.840966 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.840970 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.840974 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.840978 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.840983 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.840989 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.840993 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.841002 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.841008 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.841014 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.841022 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.841029 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.841034 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.841040 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.841046 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.841052 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.841058 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.841063 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.841069 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.841074 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-09 19:10:03.841080 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.841085 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-09 19:10:03.841090 | controller | --decode' 2026-03-09 19:10:03.841094 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-09 19:10:03.841098 | controller | CERTIFICATE-----' 2026-03-09 19:10:03.841103 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.841107 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.841111 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.841115 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.841120 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.841124 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.841130 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.841135 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.841139 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.841143 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.841165 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.841170 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.841174 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.841178 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.841191 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.841196 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.841200 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.841204 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.841209 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.841213 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.841217 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.841222 | controller | CERTIFICATE-----''' 2026-03-09 19:10:03.841226 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-09 19:10:03.841230 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.841235 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.841239 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.841243 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.841247 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.841252 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.841256 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.841260 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.841268 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.841273 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.841277 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.841281 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.841286 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.841290 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.841294 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.841299 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.841305 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.841309 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.841313 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.841318 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.841322 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.841332 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.841336 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.841340 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.841346 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.841351 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.841355 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.841359 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.841364 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.841368 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.841372 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.841377 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.841383 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.841387 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.841391 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.841396 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.841400 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.841404 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.841409 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.841414 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.841419 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.841423 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.841427 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.841432 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-09 19:10:03.841436 | controller | CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.841440 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | test 2026-03-09 19:10:03.841445 | controller | step completed 1-deploy-with-tlse' 2026-03-09 19:10:03.841449 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841454 | controller | | starting test step 2-change-public-svc-certificate' 2026-03-09 19:10:03.841458 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841462 | controller | | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-public-svc 2026-03-09 19:10:03.841467 | controller | --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-public.watcher-kuttl-default.svc'', 2026-03-09 19:10:03.841471 | controller | ''watcher-public.watcher-kuttl-default.svc.cluster.local'']}]''' 2026-03-09 19:10:03.841475 | controller | - ' ]' 2026-03-09 19:10:03.841480 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841484 | controller | | certificate.cert-manager.io/watcher-public-svc patched' 2026-03-09 19:10:03.841488 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841499 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.841503 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-09 19:10:03.841508 | controller | /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.841512 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' 2026-03-09 19:10:03.841517 | controller | | base64 --decode)' 2026-03-09 19:10:03.841521 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.841525 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-09 19:10:03.841530 | controller | - ' exit 1' 2026-03-09 19:10:03.841536 | controller | - ' fi' 2026-03-09 19:10:03.841540 | controller | - ' ]' 2026-03-09 19:10:03.841544 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841549 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.841553 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841557 | controller | | + svc_cert=''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.841562 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841566 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.841570 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841575 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.841579 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841583 | controller | | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.841588 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841592 | controller | | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.841596 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841602 | controller | | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.841607 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841611 | controller | | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.841615 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841620 | controller | | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.841624 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841628 | controller | | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.841632 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841637 | controller | | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.841641 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841645 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.841650 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841654 | controller | | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.841658 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841662 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.841667 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841671 | controller | | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.841675 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841680 | controller | | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.841684 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841688 | controller | | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.841692 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841697 | controller | | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.841703 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841707 | controller | | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.841711 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841716 | controller | | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.841720 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841724 | controller | | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.841736 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841740 | controller | | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.841746 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841751 | controller | | -----END CERTIFICATE-----''' 2026-03-09 19:10:03.841755 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841759 | controller | | ++ base64 --decode' 2026-03-09 19:10:03.841764 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841768 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.841772 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841777 | controller | | + secret_cert=''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.841781 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841785 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.841790 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841794 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' 2026-03-09 19:10:03.841798 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841803 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' 2026-03-09 19:10:03.841807 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841811 | controller | | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' 2026-03-09 19:10:03.841815 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841820 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' 2026-03-09 19:10:03.841824 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841828 | controller | | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' 2026-03-09 19:10:03.841832 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841837 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' 2026-03-09 19:10:03.841841 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841845 | controller | | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' 2026-03-09 19:10:03.841850 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841854 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.841858 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841862 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.841867 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841872 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.841877 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841881 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' 2026-03-09 19:10:03.841885 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841890 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' 2026-03-09 19:10:03.841894 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841898 | controller | | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' 2026-03-09 19:10:03.841903 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841907 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' 2026-03-09 19:10:03.841911 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841915 | controller | | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' 2026-03-09 19:10:03.841920 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841924 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' 2026-03-09 19:10:03.841928 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841933 | controller | | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' 2026-03-09 19:10:03.841937 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841941 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' 2026-03-09 19:10:03.841945 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841950 | controller | | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' 2026-03-09 19:10:03.841960 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841965 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' 2026-03-09 19:10:03.841969 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841974 | controller | | -----END CERTIFICATE-----''' 2026-03-09 19:10:03.841980 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841984 | controller | | + ''['' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.841988 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.841993 | controller | | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.841997 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842001 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' 2026-03-09 19:10:03.842006 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842012 | controller | | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' 2026-03-09 19:10:03.842018 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842024 | controller | | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' 2026-03-09 19:10:03.842033 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842039 | controller | | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' 2026-03-09 19:10:03.842043 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842047 | controller | | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' 2026-03-09 19:10:03.842054 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842058 | controller | | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' 2026-03-09 19:10:03.842062 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842066 | controller | | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' 2026-03-09 19:10:03.842071 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842075 | controller | | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.842079 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842083 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.842088 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842092 | controller | | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.842096 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842100 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' 2026-03-09 19:10:03.842104 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842109 | controller | | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' 2026-03-09 19:10:03.842113 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842117 | controller | | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' 2026-03-09 19:10:03.842121 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842126 | controller | | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' 2026-03-09 19:10:03.842130 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842134 | controller | | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' 2026-03-09 19:10:03.842138 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842142 | controller | | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' 2026-03-09 19:10:03.842161 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842166 | controller | | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' 2026-03-09 19:10:03.842170 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842174 | controller | | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' 2026-03-09 19:10:03.842178 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842183 | controller | | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' 2026-03-09 19:10:03.842187 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842191 | controller | | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.842195 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842200 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.842212 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842216 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' 2026-03-09 19:10:03.842221 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842225 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' 2026-03-09 19:10:03.842231 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842236 | controller | | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' 2026-03-09 19:10:03.842240 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842244 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' 2026-03-09 19:10:03.842249 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842253 | controller | | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' 2026-03-09 19:10:03.842257 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842261 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' 2026-03-09 19:10:03.842266 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842270 | controller | | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' 2026-03-09 19:10:03.842274 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842278 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.842283 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842287 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.842291 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842296 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.842300 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842304 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' 2026-03-09 19:10:03.842308 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842313 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' 2026-03-09 19:10:03.842317 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842321 | controller | | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' 2026-03-09 19:10:03.842327 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842331 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' 2026-03-09 19:10:03.842336 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842340 | controller | | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' 2026-03-09 19:10:03.842344 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842351 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' 2026-03-09 19:10:03.842355 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842360 | controller | | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' 2026-03-09 19:10:03.842364 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842368 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' 2026-03-09 19:10:03.842372 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842377 | controller | | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' 2026-03-09 19:10:03.842381 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842385 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' 2026-03-09 19:10:03.842390 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842396 | controller | | -----END CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.842400 | controller | - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842404 | controller | | + exit 1' 2026-03-09 19:10:03.842408 | controller | - ' logger.go:42: 19:02:37 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842413 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.842417 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-09 19:10:03.842421 | controller | /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.842426 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' 2026-03-09 19:10:03.842430 | controller | | base64 --decode)' 2026-03-09 19:10:03.842436 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.842440 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-09 19:10:03.842444 | controller | - ' exit 1' 2026-03-09 19:10:03.842456 | controller | - ' fi' 2026-03-09 19:10:03.842461 | controller | - ' ]' 2026-03-09 19:10:03.842465 | controller | - ' logger.go:42: 19:02:37 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842469 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.842474 | controller | - ' logger.go:42: 19:02:38 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842478 | controller | | error: unable to upgrade connection: container not found ("watcher-api")' 2026-03-09 19:10:03.842482 | controller | - ' logger.go:42: 19:02:38 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842487 | controller | | + svc_cert=' 2026-03-09 19:10:03.842491 | controller | - ' logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842495 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.842500 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-09 19:10:03.842504 | controller | /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.842508 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' 2026-03-09 19:10:03.842512 | controller | | base64 --decode)' 2026-03-09 19:10:03.842517 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.842521 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-09 19:10:03.842525 | controller | - ' exit 1' 2026-03-09 19:10:03.842530 | controller | - ' fi' 2026-03-09 19:10:03.842534 | controller | - ' ]' 2026-03-09 19:10:03.842538 | controller | - ' logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842542 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.842547 | controller | - ' logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842551 | controller | | error: unable to upgrade connection: container not found ("watcher-api")' 2026-03-09 19:10:03.842555 | controller | - ' logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842560 | controller | | + svc_cert=' 2026-03-09 19:10:03.842564 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842568 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.842572 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-09 19:10:03.842577 | controller | /etc/pki/tls/certs/public.crt)' 2026-03-09 19:10:03.842581 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' 2026-03-09 19:10:03.842585 | controller | | base64 --decode)' 2026-03-09 19:10:03.842590 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.842596 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-09 19:10:03.842600 | controller | - ' exit 1' 2026-03-09 19:10:03.842604 | controller | - ' fi' 2026-03-09 19:10:03.842609 | controller | - ' ]' 2026-03-09 19:10:03.842613 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842617 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-09 19:10:03.842621 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842626 | controller | | + svc_cert=''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.842630 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842634 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.842639 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842643 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' 2026-03-09 19:10:03.842647 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842651 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' 2026-03-09 19:10:03.842656 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842660 | controller | | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' 2026-03-09 19:10:03.842664 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842668 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' 2026-03-09 19:10:03.842673 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842677 | controller | | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' 2026-03-09 19:10:03.842683 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842687 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' 2026-03-09 19:10:03.842692 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842696 | controller | | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' 2026-03-09 19:10:03.842700 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842705 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.842715 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842720 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.842724 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842728 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.842733 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842737 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' 2026-03-09 19:10:03.842741 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842746 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' 2026-03-09 19:10:03.842750 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842754 | controller | | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' 2026-03-09 19:10:03.842758 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842762 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' 2026-03-09 19:10:03.842769 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842773 | controller | | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' 2026-03-09 19:10:03.842777 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842782 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' 2026-03-09 19:10:03.842786 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842790 | controller | | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' 2026-03-09 19:10:03.842795 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842799 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' 2026-03-09 19:10:03.842803 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842807 | controller | | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' 2026-03-09 19:10:03.842812 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842816 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' 2026-03-09 19:10:03.842820 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842824 | controller | | -----END CERTIFICATE-----''' 2026-03-09 19:10:03.842829 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842833 | controller | | ++ base64 --decode' 2026-03-09 19:10:03.842837 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842842 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.842846 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842850 | controller | | + secret_cert=''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.842854 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842859 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.842865 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842869 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' 2026-03-09 19:10:03.842873 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842878 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' 2026-03-09 19:10:03.842882 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842886 | controller | | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' 2026-03-09 19:10:03.842890 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842895 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' 2026-03-09 19:10:03.842899 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842903 | controller | | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' 2026-03-09 19:10:03.842907 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842912 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' 2026-03-09 19:10:03.842916 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842920 | controller | | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' 2026-03-09 19:10:03.842926 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842933 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.842944 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842949 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.842953 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842958 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.842962 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842966 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' 2026-03-09 19:10:03.842971 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842977 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' 2026-03-09 19:10:03.842985 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.842997 | controller | | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' 2026-03-09 19:10:03.843009 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843016 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' 2026-03-09 19:10:03.843021 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843027 | controller | | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' 2026-03-09 19:10:03.843033 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843039 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' 2026-03-09 19:10:03.843044 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843050 | controller | | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' 2026-03-09 19:10:03.843056 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843062 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' 2026-03-09 19:10:03.843068 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843074 | controller | | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' 2026-03-09 19:10:03.843079 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843085 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' 2026-03-09 19:10:03.843091 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843096 | controller | | -----END CERTIFICATE-----''' 2026-03-09 19:10:03.843102 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843107 | controller | | + ''['' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.843113 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843118 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.843123 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843127 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' 2026-03-09 19:10:03.843132 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843136 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' 2026-03-09 19:10:03.843141 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843159 | controller | | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' 2026-03-09 19:10:03.843170 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843175 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' 2026-03-09 19:10:03.843179 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843183 | controller | | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' 2026-03-09 19:10:03.843187 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843192 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' 2026-03-09 19:10:03.843196 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843200 | controller | | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' 2026-03-09 19:10:03.843204 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843209 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.843213 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843217 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.843221 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843225 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.843240 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843245 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' 2026-03-09 19:10:03.843250 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843254 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' 2026-03-09 19:10:03.843258 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843265 | controller | | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' 2026-03-09 19:10:03.843270 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843274 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' 2026-03-09 19:10:03.843278 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843282 | controller | | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' 2026-03-09 19:10:03.843286 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843291 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' 2026-03-09 19:10:03.843295 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843299 | controller | | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' 2026-03-09 19:10:03.843303 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843308 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' 2026-03-09 19:10:03.843312 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843316 | controller | | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' 2026-03-09 19:10:03.843320 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843325 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' 2026-03-09 19:10:03.843331 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843335 | controller | | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.843341 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843345 | controller | | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.843351 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843355 | controller | | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' 2026-03-09 19:10:03.843359 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843364 | controller | | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' 2026-03-09 19:10:03.843368 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843372 | controller | | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' 2026-03-09 19:10:03.843376 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843381 | controller | | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' 2026-03-09 19:10:03.843385 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843389 | controller | | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' 2026-03-09 19:10:03.843393 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843398 | controller | | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' 2026-03-09 19:10:03.843402 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843406 | controller | | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' 2026-03-09 19:10:03.843410 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843414 | controller | | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' 2026-03-09 19:10:03.843419 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843423 | controller | | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' 2026-03-09 19:10:03.843427 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843431 | controller | | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' 2026-03-09 19:10:03.843436 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843440 | controller | | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' 2026-03-09 19:10:03.843444 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843448 | controller | | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' 2026-03-09 19:10:03.843453 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843464 | controller | | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' 2026-03-09 19:10:03.843468 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843473 | controller | | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' 2026-03-09 19:10:03.843477 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843481 | controller | | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' 2026-03-09 19:10:03.843486 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843490 | controller | | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' 2026-03-09 19:10:03.843494 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843498 | controller | | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' 2026-03-09 19:10:03.843502 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843510 | controller | | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' 2026-03-09 19:10:03.843514 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843518 | controller | | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' 2026-03-09 19:10:03.843523 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843527 | controller | | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' 2026-03-09 19:10:03.843531 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843535 | controller | | -----END CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.843540 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-09 19:10:03.843544 | controller | | test step completed 2-change-public-svc-certificate' 2026-03-09 19:10:03.843548 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843553 | controller | | starting test step 3-change-internal-svc-certificate' 2026-03-09 19:10:03.843558 | controller | - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843564 | controller | | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc 2026-03-09 19:10:03.843570 | controller | --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-internal.watcher-kuttl-default.svc'', 2026-03-09 19:10:03.843576 | controller | ''watcher-internal.watcher-kuttl-default.svc.cluster.local'']}]''' 2026-03-09 19:10:03.843582 | controller | - ' ]' 2026-03-09 19:10:03.843599 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843606 | controller | | certificate.cert-manager.io/watcher-internal-svc patched' 2026-03-09 19:10:03.843612 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843618 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.843624 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-09 19:10:03.843630 | controller | /etc/pki/tls/certs/internal.crt)' 2026-03-09 19:10:03.843636 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o 2026-03-09 19:10:03.843642 | controller | jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-09 19:10:03.843647 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-09 19:10:03.843657 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-09 19:10:03.843663 | controller | - ' exit 1' 2026-03-09 19:10:03.843669 | controller | - ' fi' 2026-03-09 19:10:03.843675 | controller | - ' ]' 2026-03-09 19:10:03.843681 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843686 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-09 19:10:03.843692 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843698 | controller | | + svc_cert=''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.843703 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843709 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.843715 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843721 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.843727 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843732 | controller | | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.843741 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843747 | controller | | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.843753 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843758 | controller | | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.843775 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843782 | controller | | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.843787 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843792 | controller | | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.843796 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843800 | controller | | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.843804 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843809 | controller | | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.843813 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843817 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.843821 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843826 | controller | | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.843830 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843834 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.843838 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843843 | controller | | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.843850 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843854 | controller | | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.843858 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843863 | controller | | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.843867 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843871 | controller | | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.843875 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843880 | controller | | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.843884 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843888 | controller | | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.843892 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843897 | controller | | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.843901 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843905 | controller | | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.843909 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843914 | controller | | -----END CERTIFICATE-----''' 2026-03-09 19:10:03.843918 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843924 | controller | | ++ base64 --decode' 2026-03-09 19:10:03.843929 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843933 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-09 19:10:03.843937 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843941 | controller | | + secret_cert=''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.843946 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843950 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.843954 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843958 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.843963 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843967 | controller | | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.843971 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843975 | controller | | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.843989 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.843993 | controller | | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.843997 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844009 | controller | | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.844013 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844018 | controller | | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.844022 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844026 | controller | | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.844031 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844035 | controller | | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.844039 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844043 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.844048 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844052 | controller | | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.844056 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844060 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.844065 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844069 | controller | | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.844073 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844077 | controller | | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.844082 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844086 | controller | | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.844090 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844096 | controller | | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.844100 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844105 | controller | | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.844109 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844113 | controller | | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.844118 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844122 | controller | | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.844126 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844130 | controller | | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.844135 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844139 | controller | | -----END CERTIFICATE-----''' 2026-03-09 19:10:03.844163 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844171 | controller | | + ''['' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.844175 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844179 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.844184 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844188 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.844192 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844196 | controller | | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.844201 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844205 | controller | | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.844209 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844213 | controller | | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.844218 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844222 | controller | | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.844226 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844230 | controller | | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.844235 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844239 | controller | | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.844250 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844257 | controller | | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.844262 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844266 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.844270 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844275 | controller | | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.844279 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844283 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.844289 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844294 | controller | | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.844299 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844304 | controller | | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.844308 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844312 | controller | | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.844316 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844321 | controller | | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.844325 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844329 | controller | | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.844333 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844338 | controller | | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.844342 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844346 | controller | | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.844350 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844355 | controller | | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.844359 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844363 | controller | | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-09 19:10:03.844367 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844372 | controller | | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' 2026-03-09 19:10:03.844376 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844380 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' 2026-03-09 19:10:03.844384 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844389 | controller | | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' 2026-03-09 19:10:03.844393 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844397 | controller | | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' 2026-03-09 19:10:03.844401 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844406 | controller | | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' 2026-03-09 19:10:03.844410 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844414 | controller | | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' 2026-03-09 19:10:03.844419 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844423 | controller | | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' 2026-03-09 19:10:03.844427 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844431 | controller | | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' 2026-03-09 19:10:03.844436 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844440 | controller | | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-09 19:10:03.844446 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844450 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' 2026-03-09 19:10:03.844454 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844459 | controller | | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-09 19:10:03.844463 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844474 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' 2026-03-09 19:10:03.844479 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844483 | controller | | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' 2026-03-09 19:10:03.844487 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844492 | controller | | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' 2026-03-09 19:10:03.844496 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844500 | controller | | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' 2026-03-09 19:10:03.844504 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844509 | controller | | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' 2026-03-09 19:10:03.844513 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844517 | controller | | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' 2026-03-09 19:10:03.844521 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844526 | controller | | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' 2026-03-09 19:10:03.844530 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844534 | controller | | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' 2026-03-09 19:10:03.844538 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844543 | controller | | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' 2026-03-09 19:10:03.844547 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844551 | controller | | -----END CERTIFICATE-----'' '']''' 2026-03-09 19:10:03.844555 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-09 19:10:03.844560 | controller | | test step completed 3-change-internal-svc-certificate' 2026-03-09 19:10:03.844564 | controller | - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/4-cleanup-watcher | starting 2026-03-09 19:10:03.844568 | controller | test step 4-cleanup-watcher' 2026-03-09 19:10:03.844574 | controller | - ' logger.go:42: 19:02:47 | watcher-tls-certs-change/4-cleanup-watcher | test 2026-03-09 19:10:03.844579 | controller | step completed 4-cleanup-watcher' 2026-03-09 19:10:03.844583 | controller | - ' logger.go:42: 19:02:47 | watcher-tls-certs-change/5-clenaup-certs | starting 2026-03-09 19:10:03.844587 | controller | test step 5-clenaup-certs' 2026-03-09 19:10:03.844592 | controller | - ' logger.go:42: 19:02:47 | watcher-tls-certs-change/5-clenaup-certs | test step 2026-03-09 19:10:03.844596 | controller | completed 5-clenaup-certs' 2026-03-09 19:10:03.844600 | controller | - ' logger.go:42: 19:02:47 | watcher-tls-certs-change | skipping kubernetes event 2026-03-09 19:10:03.844604 | controller | logging' 2026-03-09 19:10:03.844609 | controller | - === CONT kuttl/harness/watcher-tls 2026-03-09 19:10:03.844613 | controller | - ' logger.go:42: 19:02:47 | watcher-tls | Skipping creation of user-supplied namespace: 2026-03-09 19:10:03.844617 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.844623 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/0-cleanup-watcher | starting test step 2026-03-09 19:10:03.844627 | controller | 0-cleanup-watcher' 2026-03-09 19:10:03.844632 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/0-cleanup-watcher | test step completed 2026-03-09 19:10:03.844636 | controller | 0-cleanup-watcher' 2026-03-09 19:10:03.844640 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | starting test step 2026-03-09 19:10:03.844644 | controller | 1-deploy-with-tlse' 2026-03-09 19:10:03.844649 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc 2026-03-09 19:10:03.844653 | controller | created' 2026-03-09 19:10:03.844657 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc 2026-03-09 19:10:03.844662 | controller | created' 2026-03-09 19:10:03.844666 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.844670 | controller | created' 2026-03-09 19:10:03.844674 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-09 19:10:03.844679 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.844683 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.844687 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.844692 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.844696 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.844700 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.844704 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.844709 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.844713 | controller | ]' 2026-03-09 19:10:03.844717 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-09 19:10:03.844722 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.844733 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-09 19:10:03.844737 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-09 19:10:03.844742 | controller | ]' 2026-03-09 19:10:03.844746 | controller | - ' # check that both endpoints have https set' 2026-03-09 19:10:03.844750 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.844755 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-09 19:10:03.844759 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.844763 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.844767 | controller | == "" ]; then' 2026-03-09 19:10:03.844772 | controller | - ' exit 0' 2026-03-09 19:10:03.844776 | controller | - ' fi' 2026-03-09 19:10:03.844780 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.844785 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.844789 | controller | - ' counter=0' 2026-03-09 19:10:03.844793 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.844799 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.844804 | controller | - ' echo ${i}' 2026-03-09 19:10:03.844808 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.844812 | controller | - ' fi' 2026-03-09 19:10:03.844816 | controller | - ' done' 2026-03-09 19:10:03.844821 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.844825 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.844831 | controller | - ' exit 1' 2026-03-09 19:10:03.844835 | controller | - ' else' 2026-03-09 19:10:03.844839 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.844844 | controller | - ' fi' 2026-03-09 19:10:03.844848 | controller | - ' ]' 2026-03-09 19:10:03.844852 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.844856 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.844861 | controller | - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.844865 | controller | - ' logger.go:42: 19:02:49 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.844869 | controller | - ' logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-09 19:10:03.844873 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.844878 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.844882 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.844886 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.844891 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.844895 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.844899 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.844903 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.844908 | controller | ]' 2026-03-09 19:10:03.844912 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-09 19:10:03.844916 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.844921 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-09 19:10:03.844925 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-09 19:10:03.844929 | controller | ]' 2026-03-09 19:10:03.844933 | controller | - ' # check that both endpoints have https set' 2026-03-09 19:10:03.844938 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.844942 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-09 19:10:03.844946 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.844950 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.844958 | controller | == "" ]; then' 2026-03-09 19:10:03.844962 | controller | - ' exit 0' 2026-03-09 19:10:03.844966 | controller | - ' fi' 2026-03-09 19:10:03.844971 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.844975 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.844979 | controller | - ' counter=0' 2026-03-09 19:10:03.844983 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.844988 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.844992 | controller | - ' echo ${i}' 2026-03-09 19:10:03.844996 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.845001 | controller | - ' fi' 2026-03-09 19:10:03.845005 | controller | - ' done' 2026-03-09 19:10:03.845009 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.845013 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.845018 | controller | - ' exit 1' 2026-03-09 19:10:03.845022 | controller | - ' else' 2026-03-09 19:10:03.845026 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.845030 | controller | - ' fi' 2026-03-09 19:10:03.845036 | controller | - ' ]' 2026-03-09 19:10:03.845041 | controller | - ' logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845045 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.845056 | controller | - ' logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.845061 | controller | - ' logger.go:42: 19:02:53 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.845065 | controller | - ' logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-09 19:10:03.845070 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.845074 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.845078 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.845083 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.845087 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.845091 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.845095 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.845100 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.845104 | controller | ]' 2026-03-09 19:10:03.845108 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-09 19:10:03.845112 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.845117 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-09 19:10:03.845121 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-09 19:10:03.845125 | controller | ]' 2026-03-09 19:10:03.845130 | controller | - ' # check that both endpoints have https set' 2026-03-09 19:10:03.845134 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.845138 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-09 19:10:03.845142 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.845161 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.845165 | controller | == "" ]; then' 2026-03-09 19:10:03.845170 | controller | - ' exit 0' 2026-03-09 19:10:03.845174 | controller | - ' fi' 2026-03-09 19:10:03.845180 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.845185 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.845189 | controller | - ' counter=0' 2026-03-09 19:10:03.845193 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.845198 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.845202 | controller | - ' echo ${i}' 2026-03-09 19:10:03.845206 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.845210 | controller | - ' fi' 2026-03-09 19:10:03.845215 | controller | - ' done' 2026-03-09 19:10:03.845219 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.845223 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.845227 | controller | - ' exit 1' 2026-03-09 19:10:03.845232 | controller | - ' else' 2026-03-09 19:10:03.845236 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.845240 | controller | - ' fi' 2026-03-09 19:10:03.845244 | controller | - ' ]' 2026-03-09 19:10:03.845249 | controller | - ' logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.845253 | controller | - ' logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845260 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.845264 | controller | - ' logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.845270 | controller | - ' logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' 2026-03-09 19:10:03.845274 | controller | - ' logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845278 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.845283 | controller | - ' logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' 2026-03-09 19:10:03.845287 | controller | - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.845291 | controller | - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845295 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.845299 | controller | - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.845304 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.845308 | controller | - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845312 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.845316 | controller | - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + ''['' -n '''' '']''' 2026-03-09 19:10:03.845321 | controller | - ' logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-09 19:10:03.845325 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.845329 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.845333 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.845345 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.845349 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.845354 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.845358 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.845362 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.845367 | controller | ]' 2026-03-09 19:10:03.845371 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-09 19:10:03.845375 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.845379 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-09 19:10:03.845384 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-09 19:10:03.845388 | controller | ]' 2026-03-09 19:10:03.845392 | controller | - ' # check that both endpoints have https set' 2026-03-09 19:10:03.845397 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.845401 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-09 19:10:03.845405 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.845409 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.845414 | controller | == "" ]; then' 2026-03-09 19:10:03.845418 | controller | - ' exit 0' 2026-03-09 19:10:03.845422 | controller | - ' fi' 2026-03-09 19:10:03.845426 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.845431 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.845435 | controller | - ' counter=0' 2026-03-09 19:10:03.845441 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.845445 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.845450 | controller | - ' echo ${i}' 2026-03-09 19:10:03.845454 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.845458 | controller | - ' fi' 2026-03-09 19:10:03.845462 | controller | - ' done' 2026-03-09 19:10:03.845467 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.845471 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.845475 | controller | - ' exit 1' 2026-03-09 19:10:03.845479 | controller | - ' else' 2026-03-09 19:10:03.845484 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.845488 | controller | - ' fi' 2026-03-09 19:10:03.845494 | controller | - ' ]' 2026-03-09 19:10:03.845499 | controller | - ' logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845505 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.845511 | controller | - ' logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.845516 | controller | - ' logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.845522 | controller | - ' logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845528 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.845534 | controller | - ' logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' 2026-03-09 19:10:03.845540 | controller | - ' logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' 2026-03-09 19:10:03.845545 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.845552 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845558 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.845562 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.845566 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.845571 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845575 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.845579 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.845583 | controller | '']''' 2026-03-09 19:10:03.845588 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-09 19:10:03.845595 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845599 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-09 19:10:03.845604 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' 2026-03-09 19:10:03.845608 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.845612 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-09 19:10:03.845617 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile 2026-03-09 19:10:03.845629 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-09 19:10:03.845634 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845638 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.845645 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.845649 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' 2026-03-09 19:10:03.845653 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845658 | controller | openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.845662 | controller | - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' 2026-03-09 19:10:03.845668 | controller | - ' logger.go:42: 19:03:06 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.845672 | controller | - ' logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-09 19:10:03.845676 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.845681 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.845685 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.845689 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.845693 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.845698 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.845702 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.845706 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.845712 | controller | ]' 2026-03-09 19:10:03.845717 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-09 19:10:03.845721 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.845725 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-09 19:10:03.845729 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-09 19:10:03.845734 | controller | ]' 2026-03-09 19:10:03.845738 | controller | - ' # check that both endpoints have https set' 2026-03-09 19:10:03.845742 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.845747 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-09 19:10:03.845751 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.845755 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.845759 | controller | == "" ]; then' 2026-03-09 19:10:03.845764 | controller | - ' exit 0' 2026-03-09 19:10:03.845768 | controller | - ' fi' 2026-03-09 19:10:03.845772 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.845777 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.845781 | controller | - ' counter=0' 2026-03-09 19:10:03.845785 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.845791 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.845796 | controller | - ' echo ${i}' 2026-03-09 19:10:03.845800 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.845804 | controller | - ' fi' 2026-03-09 19:10:03.845809 | controller | - ' done' 2026-03-09 19:10:03.845813 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.845818 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.845822 | controller | - ' exit 1' 2026-03-09 19:10:03.845826 | controller | - ' else' 2026-03-09 19:10:03.845830 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.845835 | controller | - ' fi' 2026-03-09 19:10:03.845839 | controller | - ' ]' 2026-03-09 19:10:03.845843 | controller | - ' logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845851 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.845856 | controller | - ' logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.845861 | controller | - ' logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.845865 | controller | - ' logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' 2026-03-09 19:10:03.845869 | controller | - ' logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' 2026-03-09 19:10:03.845873 | controller | - ' logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845878 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.845883 | controller | - ' logger.go:42: 19:03:11 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.845889 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845895 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.845901 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.845916 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.845922 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845926 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.845930 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.845935 | controller | '']''' 2026-03-09 19:10:03.845939 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' 2026-03-09 19:10:03.845943 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-09 19:10:03.845947 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845952 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-09 19:10:03.845956 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.845960 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile 2026-03-09 19:10:03.845964 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-09 19:10:03.845969 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.845973 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.845977 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-09 19:10:03.845981 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.845986 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' 2026-03-09 19:10:03.845990 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.845994 | controller | openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.845998 | controller | - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' 2026-03-09 19:10:03.846002 | controller | - ' logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.846007 | controller | - ' logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n 2026-03-09 19:10:03.846011 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.846015 | controller | - ' logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == '''' 2026-03-09 19:10:03.846022 | controller | '']''' 2026-03-09 19:10:03.846026 | controller | - ' logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + exit 0' 2026-03-09 19:10:03.846030 | controller | - ' logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-09 19:10:03.846035 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.846039 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.846043 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.846047 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.846052 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.846056 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.846060 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.846064 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.846069 | controller | ]' 2026-03-09 19:10:03.846073 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-09 19:10:03.846077 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.846081 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-09 19:10:03.846086 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-09 19:10:03.846090 | controller | ]' 2026-03-09 19:10:03.846094 | controller | - ' # check that both endpoints have https set' 2026-03-09 19:10:03.846098 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.846103 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-09 19:10:03.846107 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.846111 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.846115 | controller | == "" ]; then' 2026-03-09 19:10:03.846120 | controller | - ' exit 0' 2026-03-09 19:10:03.846124 | controller | - ' fi' 2026-03-09 19:10:03.846128 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.846132 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.846137 | controller | - ' counter=0' 2026-03-09 19:10:03.846141 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.846157 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.846164 | controller | - ' echo ${i}' 2026-03-09 19:10:03.846169 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.846173 | controller | - ' fi' 2026-03-09 19:10:03.846177 | controller | - ' done' 2026-03-09 19:10:03.846190 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.846196 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.846202 | controller | - ' exit 1' 2026-03-09 19:10:03.846208 | controller | - ' else' 2026-03-09 19:10:03.846213 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.846219 | controller | - ' fi' 2026-03-09 19:10:03.846225 | controller | - ' ]' 2026-03-09 19:10:03.846231 | controller | - ' logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.846237 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.846243 | controller | - ' logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.846249 | controller | - ' logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.846254 | controller | - ' logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.846265 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.846271 | controller | - ' logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' 2026-03-09 19:10:03.846277 | controller | - ' logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' 2026-03-09 19:10:03.846283 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.846287 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.846291 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.846296 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.846300 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.846304 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.846309 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.846313 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.846317 | controller | '']''' 2026-03-09 19:10:03.846321 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' 2026-03-09 19:10:03.846326 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-09 19:10:03.846330 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.846334 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-09 19:10:03.846338 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.846345 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-09 19:10:03.846350 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.846354 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.846360 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile 2026-03-09 19:10:03.846365 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-09 19:10:03.846371 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.846377 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' 2026-03-09 19:10:03.846383 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' 2026-03-09 19:10:03.846388 | controller | - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.846396 | controller | openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.846402 | controller | - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.846408 | controller | - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n 2026-03-09 19:10:03.846414 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.846419 | controller | - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == '''' 2026-03-09 19:10:03.846425 | controller | '']''' 2026-03-09 19:10:03.846431 | controller | - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + exit 0' 2026-03-09 19:10:03.846436 | controller | - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | test step completed 2026-03-09 19:10:03.846442 | controller | 1-deploy-with-tlse' 2026-03-09 19:10:03.846448 | controller | - ' logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls' 2026-03-09 19:10:03.846457 | controller | - ' logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-09 19:10:03.846462 | controller | set -euxo pipefail' 2026-03-09 19:10:03.846468 | controller | - ' oc patch oscp -n $NAMESPACE openstack --type=''json'' -p=''[{"op": "replace", 2026-03-09 19:10:03.846473 | controller | "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode", "value": "Request"}]''' 2026-03-09 19:10:03.846479 | controller | - ' ]' 2026-03-09 19:10:03.846496 | controller | - ' logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | + oc patch oscp -n watcher-kuttl-default 2026-03-09 19:10:03.846503 | controller | openstack --type=json ''-p=[{"op": "replace", "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode", 2026-03-09 19:10:03.846508 | controller | "value": "Request"}]''' 2026-03-09 19:10:03.846514 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack 2026-03-09 19:10:03.846520 | controller | patched' 2026-03-09 19:10:03.846526 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-09 19:10:03.846531 | controller | set -euxo pipefail' 2026-03-09 19:10:03.846537 | controller | - ' ' 2026-03-09 19:10:03.846543 | controller | - ' oc project ${NAMESPACE}' 2026-03-09 19:10:03.846548 | controller | - ' # Get pod names for each watcher service' 2026-03-09 19:10:03.846553 | controller | - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.846557 | controller | - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.846561 | controller | - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.846565 | controller | - ' ' 2026-03-09 19:10:03.846570 | controller | - ' # Verify memcached mTLS config parameters in watcher-api config' 2026-03-09 19:10:03.846577 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.846581 | controller | - ' echo "Checking watcher-api config..."' 2026-03-09 19:10:03.846585 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846590 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.846594 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846598 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.846602 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846607 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.846611 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846615 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.846619 | controller | - ' ' 2026-03-09 19:10:03.846624 | controller | - ' # Verify mTLS config parameters in memcached backend config' 2026-03-09 19:10:03.846628 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846632 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.846637 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846641 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.846645 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846649 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.846653 | controller | - ' else' 2026-03-09 19:10:03.846658 | controller | - ' exit 1' 2026-03-09 19:10:03.846664 | controller | - ' fi' 2026-03-09 19:10:03.846668 | controller | - ' ' 2026-03-09 19:10:03.846673 | controller | - ' # Verify memcached mTLS config parameters in watcher-applier config' 2026-03-09 19:10:03.846677 | controller | - ' if [ -n "${APPLIERPOD}" ]; then' 2026-03-09 19:10:03.846681 | controller | - ' echo "Checking watcher-applier config..."' 2026-03-09 19:10:03.846685 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846690 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.846694 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846698 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.846702 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846707 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.846711 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846715 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.846719 | controller | - ' ' 2026-03-09 19:10:03.846724 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846728 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.846732 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846736 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.846740 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846745 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.846749 | controller | - ' else' 2026-03-09 19:10:03.846762 | controller | - ' exit 1' 2026-03-09 19:10:03.846766 | controller | - ' fi' 2026-03-09 19:10:03.846770 | controller | - ' ' 2026-03-09 19:10:03.846775 | controller | - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' 2026-03-09 19:10:03.846779 | controller | - ' if [ -n "${DECISIONENGINEPOD}" ]; then' 2026-03-09 19:10:03.846783 | controller | - ' echo "Checking watcher-decision-engine config..."' 2026-03-09 19:10:03.846788 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846792 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.846796 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846801 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.846805 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846809 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.846813 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846818 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.846822 | controller | - ' ' 2026-03-09 19:10:03.846826 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846830 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.846835 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846841 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.846845 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.846849 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.846853 | controller | - ' else' 2026-03-09 19:10:03.846858 | controller | - ' exit 1' 2026-03-09 19:10:03.846862 | controller | - ' fi' 2026-03-09 19:10:03.846866 | controller | - ' ]' 2026-03-09 19:10:03.846870 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.846875 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" 2026-03-09 19:10:03.846879 | controller | on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.846883 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.846888 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.846892 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.846896 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier 2026-03-09 19:10:03.846900 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.846905 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' 2026-03-09 19:10:03.846909 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.846913 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.846917 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.846922 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 2026-03-09 19:10:03.846926 | controller | '']''' 2026-03-09 19:10:03.846930 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api 2026-03-09 19:10:03.846935 | controller | config...''' 2026-03-09 19:10:03.846941 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | Checking watcher-api config...' 2026-03-09 19:10:03.846945 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.846949 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.846954 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.846958 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.846962 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.846967 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.846971 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.846975 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.846980 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.846992 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.846996 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.847000 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.847005 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-09 19:10:03.847013 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-09 19:10:03.847020 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.847026 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.847035 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.847041 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.847045 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.847050 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.847054 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.847058 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.847062 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-09 19:10:03.847067 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-09 19:10:03.847071 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.847075 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.847079 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.847084 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.847088 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.847092 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.847096 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.847101 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.847105 | controller | - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.847109 | controller | - ' logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-09 19:10:03.847113 | controller | set -euxo pipefail' 2026-03-09 19:10:03.847118 | controller | - ' ' 2026-03-09 19:10:03.847122 | controller | - ' oc project ${NAMESPACE}' 2026-03-09 19:10:03.847126 | controller | - ' # Get pod names for each watcher service' 2026-03-09 19:10:03.847131 | controller | - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.847135 | controller | - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.847139 | controller | - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.847143 | controller | - ' ' 2026-03-09 19:10:03.847164 | controller | - ' # Verify memcached mTLS config parameters in watcher-api config' 2026-03-09 19:10:03.847168 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.847172 | controller | - ' echo "Checking watcher-api config..."' 2026-03-09 19:10:03.847177 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847181 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.847185 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847192 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847196 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847200 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847207 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847211 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.847215 | controller | - ' ' 2026-03-09 19:10:03.847220 | controller | - ' # Verify mTLS config parameters in memcached backend config' 2026-03-09 19:10:03.847224 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847237 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.847242 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847246 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847250 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847255 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847259 | controller | - ' else' 2026-03-09 19:10:03.847263 | controller | - ' exit 1' 2026-03-09 19:10:03.847268 | controller | - ' fi' 2026-03-09 19:10:03.847272 | controller | - ' ' 2026-03-09 19:10:03.847276 | controller | - ' # Verify memcached mTLS config parameters in watcher-applier config' 2026-03-09 19:10:03.847280 | controller | - ' if [ -n "${APPLIERPOD}" ]; then' 2026-03-09 19:10:03.847285 | controller | - ' echo "Checking watcher-applier config..."' 2026-03-09 19:10:03.847289 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847293 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.847298 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847302 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847306 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847310 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847315 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847319 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.847323 | controller | - ' ' 2026-03-09 19:10:03.847327 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847332 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.847336 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847340 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847345 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847351 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847355 | controller | - ' else' 2026-03-09 19:10:03.847360 | controller | - ' exit 1' 2026-03-09 19:10:03.847364 | controller | - ' fi' 2026-03-09 19:10:03.847368 | controller | - ' ' 2026-03-09 19:10:03.847372 | controller | - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' 2026-03-09 19:10:03.847378 | controller | - ' if [ -n "${DECISIONENGINEPOD}" ]; then' 2026-03-09 19:10:03.847385 | controller | - ' echo "Checking watcher-decision-engine config..."' 2026-03-09 19:10:03.847390 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847394 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.847398 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847403 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847407 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847411 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847416 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847420 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.847424 | controller | - ' ' 2026-03-09 19:10:03.847428 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847433 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.847437 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847441 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847446 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847450 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847454 | controller | - ' else' 2026-03-09 19:10:03.847458 | controller | - ' exit 1' 2026-03-09 19:10:03.847463 | controller | - ' fi' 2026-03-09 19:10:03.847467 | controller | - ' ]' 2026-03-09 19:10:03.847471 | controller | - ' logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.847476 | controller | - ' logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" 2026-03-09 19:10:03.847480 | controller | on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.847490 | controller | - ' logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.847495 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.847499 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.847504 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier 2026-03-09 19:10:03.847508 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.847512 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' 2026-03-09 19:10:03.847517 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.847521 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.847525 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.847529 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 2026-03-09 19:10:03.847534 | controller | '']''' 2026-03-09 19:10:03.847538 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api 2026-03-09 19:10:03.847542 | controller | config...''' 2026-03-09 19:10:03.847546 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | Checking watcher-api config...' 2026-03-09 19:10:03.847551 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.847557 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.847561 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.847565 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.847570 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | error: cannot exec into 2026-03-09 19:10:03.847574 | controller | a container in a completed pod; current phase is Failed' 2026-03-09 19:10:03.847578 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ echo' 2026-03-09 19:10:03.847582 | controller | - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.847613 | controller | - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-09 19:10:03.847617 | controller | set -euxo pipefail' 2026-03-09 19:10:03.847622 | controller | - ' ' 2026-03-09 19:10:03.847626 | controller | - ' oc project ${NAMESPACE}' 2026-03-09 19:10:03.847630 | controller | - ' # Get pod names for each watcher service' 2026-03-09 19:10:03.847635 | controller | - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.847639 | controller | - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.847643 | controller | - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.847647 | controller | - ' ' 2026-03-09 19:10:03.847652 | controller | - ' # Verify memcached mTLS config parameters in watcher-api config' 2026-03-09 19:10:03.847656 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.847660 | controller | - ' echo "Checking watcher-api config..."' 2026-03-09 19:10:03.847664 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847669 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.847673 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847677 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847681 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847686 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847690 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847694 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.847698 | controller | - ' ' 2026-03-09 19:10:03.847703 | controller | - ' # Verify mTLS config parameters in memcached backend config' 2026-03-09 19:10:03.847707 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847711 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.847716 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847720 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847724 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847728 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847733 | controller | - ' else' 2026-03-09 19:10:03.847737 | controller | - ' exit 1' 2026-03-09 19:10:03.847741 | controller | - ' fi' 2026-03-09 19:10:03.847745 | controller | - ' ' 2026-03-09 19:10:03.847749 | controller | - ' # Verify memcached mTLS config parameters in watcher-applier config' 2026-03-09 19:10:03.847754 | controller | - ' if [ -n "${APPLIERPOD}" ]; then' 2026-03-09 19:10:03.847758 | controller | - ' echo "Checking watcher-applier config..."' 2026-03-09 19:10:03.847772 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847776 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.847781 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847785 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847789 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847794 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847798 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847802 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.847807 | controller | - ' ' 2026-03-09 19:10:03.847811 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847815 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.847819 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847824 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847828 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847832 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847836 | controller | - ' else' 2026-03-09 19:10:03.847841 | controller | - ' exit 1' 2026-03-09 19:10:03.847845 | controller | - ' fi' 2026-03-09 19:10:03.847849 | controller | - ' ' 2026-03-09 19:10:03.847854 | controller | - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' 2026-03-09 19:10:03.847858 | controller | - ' if [ -n "${DECISIONENGINEPOD}" ]; then' 2026-03-09 19:10:03.847862 | controller | - ' echo "Checking watcher-decision-engine config..."' 2026-03-09 19:10:03.847867 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847871 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.847877 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847881 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847885 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847890 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847894 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847898 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.847903 | controller | - ' ' 2026-03-09 19:10:03.847907 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847911 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.847916 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847920 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.847924 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.847930 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.847934 | controller | - ' else' 2026-03-09 19:10:03.847939 | controller | - ' exit 1' 2026-03-09 19:10:03.847943 | controller | - ' fi' 2026-03-09 19:10:03.847947 | controller | - ' ]' 2026-03-09 19:10:03.847952 | controller | - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.847956 | controller | - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" 2026-03-09 19:10:03.847960 | controller | on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.847964 | controller | - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.847969 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.847973 | controller | - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.847977 | controller | - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier 2026-03-09 19:10:03.847982 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.847986 | controller | - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' 2026-03-09 19:10:03.847990 | controller | - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.847995 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.847999 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.848011 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 2026-03-09 19:10:03.848018 | controller | '']''' 2026-03-09 19:10:03.848024 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api 2026-03-09 19:10:03.848030 | controller | config...''' 2026-03-09 19:10:03.848036 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | Checking watcher-api config...' 2026-03-09 19:10:03.848041 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.848046 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.848050 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.848054 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.848059 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | error: unable to upgrade 2026-03-09 19:10:03.848063 | controller | connection: container not found ("watcher-api")' 2026-03-09 19:10:03.848067 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | ++ echo' 2026-03-09 19:10:03.848071 | controller | - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.848076 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-09 19:10:03.848080 | controller | set -euxo pipefail' 2026-03-09 19:10:03.848084 | controller | - ' ' 2026-03-09 19:10:03.848088 | controller | - ' oc project ${NAMESPACE}' 2026-03-09 19:10:03.848093 | controller | - ' # Get pod names for each watcher service' 2026-03-09 19:10:03.848097 | controller | - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.848101 | controller | - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.848106 | controller | - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.848110 | controller | - ' ' 2026-03-09 19:10:03.848114 | controller | - ' # Verify memcached mTLS config parameters in watcher-api config' 2026-03-09 19:10:03.848118 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.848123 | controller | - ' echo "Checking watcher-api config..."' 2026-03-09 19:10:03.848127 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848134 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.848140 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848157 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.848163 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848168 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.848172 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848176 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.848180 | controller | - ' ' 2026-03-09 19:10:03.848185 | controller | - ' # Verify mTLS config parameters in memcached backend config' 2026-03-09 19:10:03.848189 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848193 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.848198 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848202 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.848208 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848212 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.848216 | controller | - ' else' 2026-03-09 19:10:03.848222 | controller | - ' exit 1' 2026-03-09 19:10:03.848227 | controller | - ' fi' 2026-03-09 19:10:03.848231 | controller | - ' ' 2026-03-09 19:10:03.848235 | controller | - ' # Verify memcached mTLS config parameters in watcher-applier config' 2026-03-09 19:10:03.848240 | controller | - ' if [ -n "${APPLIERPOD}" ]; then' 2026-03-09 19:10:03.848244 | controller | - ' echo "Checking watcher-applier config..."' 2026-03-09 19:10:03.848248 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848252 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.848257 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848261 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.848265 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848269 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.848274 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848278 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.848282 | controller | - ' ' 2026-03-09 19:10:03.848295 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848300 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.848304 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848308 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.848313 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848317 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.848321 | controller | - ' else' 2026-03-09 19:10:03.848325 | controller | - ' exit 1' 2026-03-09 19:10:03.848331 | controller | - ' fi' 2026-03-09 19:10:03.848336 | controller | - ' ' 2026-03-09 19:10:03.848340 | controller | - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' 2026-03-09 19:10:03.848345 | controller | - ' if [ -n "${DECISIONENGINEPOD}" ]; then' 2026-03-09 19:10:03.848349 | controller | - ' echo "Checking watcher-decision-engine config..."' 2026-03-09 19:10:03.848353 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848357 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.848362 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848371 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.848376 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848380 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.848384 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848388 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.848393 | controller | - ' ' 2026-03-09 19:10:03.848397 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848401 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.848406 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848410 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.848414 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.848418 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.848423 | controller | - ' else' 2026-03-09 19:10:03.848427 | controller | - ' exit 1' 2026-03-09 19:10:03.848431 | controller | - ' fi' 2026-03-09 19:10:03.848435 | controller | - ' ]' 2026-03-09 19:10:03.848440 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.848444 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" 2026-03-09 19:10:03.848448 | controller | on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.848453 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.848457 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.848461 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.848465 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier 2026-03-09 19:10:03.848470 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.848474 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' 2026-03-09 19:10:03.848478 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.848482 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.848487 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.848491 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 2026-03-09 19:10:03.848495 | controller | '']''' 2026-03-09 19:10:03.848499 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api 2026-03-09 19:10:03.848505 | controller | config...''' 2026-03-09 19:10:03.848510 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | Checking watcher-api config...' 2026-03-09 19:10:03.848514 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.848518 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.848523 | controller | - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.848527 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.848531 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.848543 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.848547 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.848551 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.848556 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.848560 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.848565 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.848569 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.848573 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.848577 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.848582 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.848586 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.848590 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.848594 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.848599 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.848603 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.848607 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.848611 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.848615 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.848620 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.848624 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.848628 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.848632 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.848637 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.848641 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.848645 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.848649 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.848655 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.848676 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.848680 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.848685 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.848689 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.848693 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.848697 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.848702 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile 2026-03-09 19:10:03.848706 | controller | = /etc/pki/tls/private/mtls.key''' 2026-03-09 19:10:03.848710 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.848715 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.848719 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.848723 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.848727 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.848732 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.848742 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.848747 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.848753 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.848757 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.848762 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.848766 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.848770 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.848774 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.848779 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.848783 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.848787 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.848794 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.848799 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.848803 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.848807 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.848811 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.848816 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.848822 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.848826 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.848830 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.848835 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.848839 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.848843 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.848847 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.848851 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.848856 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.848860 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.848864 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.848868 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.848873 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.848877 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile 2026-03-09 19:10:03.848881 | controller | = /etc/pki/tls/certs/mtls-ca.crt''' 2026-03-09 19:10:03.848885 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.848890 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.848894 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.848898 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.848902 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.848907 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.848911 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.848915 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.848919 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.848930 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.848935 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.848939 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.848943 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.848948 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.848952 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.848956 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.848960 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.848966 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.848971 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.848975 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.848979 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.848983 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.848988 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.848994 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.848998 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.849002 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.849007 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.849013 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.849019 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.849025 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.849031 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.849037 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.849043 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.849047 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.849051 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.849056 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.849062 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled 2026-03-09 19:10:03.849066 | controller | = true''' 2026-03-09 19:10:03.849071 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.849075 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.849079 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.849083 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.849088 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.849092 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.849096 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.849100 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.849105 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.849109 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849113 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.849119 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.849124 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.849136 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849141 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849157 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.849164 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.849169 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849173 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849177 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.849181 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.849186 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849190 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.849197 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.849201 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.849206 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.849210 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.849214 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.849218 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.849223 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.849227 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.849231 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.849235 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.849240 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.849244 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.849248 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.849252 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' 2026-03-09 19:10:03.849257 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.849261 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.849265 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.849270 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.849274 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.849278 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.849282 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.849289 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.849293 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.849297 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849301 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.849306 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.849310 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.849314 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849318 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849323 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.849327 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.849338 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849343 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849347 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.849351 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.849355 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849360 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.849364 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.849368 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.849373 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.849386 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.849391 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.849395 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.849399 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.849403 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.849408 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.849412 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.849416 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.849420 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.849425 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.849429 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' 2026-03-09 19:10:03.849433 | controller | - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.849439 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.849445 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.849449 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.849453 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.849458 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.849462 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.849466 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.849472 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.849476 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849481 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.849485 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.849489 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.849493 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849498 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849502 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.849506 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.849510 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849515 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849519 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.849523 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.849534 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849539 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.849543 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.849547 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.849552 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.849556 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.849560 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.849564 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.849569 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.849573 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.849577 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.849581 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.849585 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.849592 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.849596 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.849601 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' 2026-03-09 19:10:03.849605 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.849609 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.849613 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.849618 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.849622 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.849626 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.849631 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.849637 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.849641 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.849645 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849649 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.849654 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.849658 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.849662 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849668 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849672 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.849676 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.849681 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849685 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849689 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.849693 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.849697 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849702 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.849706 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.849710 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.849714 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.849726 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.849731 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.849738 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.849743 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.849747 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.849751 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.849756 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.849760 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.849764 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.849769 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.849773 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0 2026-03-09 19:10:03.849777 | controller | '']''' 2026-03-09 19:10:03.849781 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier 2026-03-09 19:10:03.849786 | controller | config...''' 2026-03-09 19:10:03.849790 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | Checking watcher-applier 2026-03-09 19:10:03.849794 | controller | config...' 2026-03-09 19:10:03.849799 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.849803 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.849807 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.849811 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.849816 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.849820 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.849824 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.849829 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.849833 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.849837 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.849842 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849846 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.849850 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.849854 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.849859 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849863 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849867 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.849871 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.849876 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.849880 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.849886 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.849890 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.849894 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.849899 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.849903 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.849907 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.849911 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.849916 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.849925 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.849930 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.849936 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.849940 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.849944 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.849949 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.849953 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.849957 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile 2026-03-09 19:10:03.849961 | controller | = /etc/pki/tls/private/mtls.key''' 2026-03-09 19:10:03.849966 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.849970 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.849974 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.849979 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.849983 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.849987 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.849992 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.849996 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.850000 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850005 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.850011 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.850017 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.850023 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850029 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850035 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.850042 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.850047 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850051 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850055 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.850059 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.850064 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850068 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.850072 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.850077 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.850081 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.850085 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.850089 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.850096 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.850100 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.850104 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.850108 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.850114 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.850125 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.850129 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile 2026-03-09 19:10:03.850134 | controller | = /etc/pki/tls/certs/mtls-ca.crt''' 2026-03-09 19:10:03.850138 | controller | - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.850142 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.850164 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.850169 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.850173 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.850177 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.850182 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.850186 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.850190 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850195 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.850199 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.850205 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.850209 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850214 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850218 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.850222 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.850226 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850231 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850235 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.850239 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.850243 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850250 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.850254 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.850258 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.850263 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.850267 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.850271 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.850275 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.850280 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.850284 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.850288 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.850292 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.850297 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.850301 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled 2026-03-09 19:10:03.850305 | controller | = true''' 2026-03-09 19:10:03.850309 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.850314 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.850318 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.850328 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.850332 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.850337 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.850341 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.850346 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.850352 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850356 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.850360 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.850364 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.850369 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850373 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850377 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.850381 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.850386 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850390 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850394 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.850398 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.850403 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850407 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.850411 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.850415 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.850420 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.850424 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.850428 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.850432 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.850437 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.850441 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.850445 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.850449 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.850454 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.850458 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' 2026-03-09 19:10:03.850462 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.850466 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.850471 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.850475 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.850479 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.850485 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.850491 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.850496 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.850505 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850510 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.850514 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.850518 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.850522 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850528 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850533 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.850537 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.850541 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850545 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850550 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.850554 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.850558 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850562 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.850568 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.850572 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.850577 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.850581 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.850585 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.850589 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.850594 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.850598 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.850602 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.850607 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.850611 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.850615 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' 2026-03-09 19:10:03.850619 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.850625 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.850629 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.850634 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.850640 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.850644 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.850648 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.850653 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.850657 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850661 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.850665 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.850670 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.850674 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850678 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850682 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.850693 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.850697 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850702 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850706 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.850710 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.850714 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850719 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.850723 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.850727 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.850731 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.850736 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.850740 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.850744 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.850748 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.850753 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.850757 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.850761 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.850765 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.850770 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' 2026-03-09 19:10:03.850777 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.850783 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.850787 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.850791 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.850795 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.850800 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.850804 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.850808 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.850812 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850817 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.850821 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.850825 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.850829 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850834 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850838 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.850842 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.850846 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.850851 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.850855 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.850859 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.850863 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.850873 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.850877 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.850882 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.850886 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.850890 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.850894 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.850899 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.850903 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.850907 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.850911 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.850916 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.850922 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.850926 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.850930 | controller | '']''' 2026-03-09 19:10:03.850935 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine 2026-03-09 19:10:03.850939 | controller | config...''' 2026-03-09 19:10:03.850943 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine 2026-03-09 19:10:03.850947 | controller | config...' 2026-03-09 19:10:03.850953 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.850958 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.850962 | controller | - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.850966 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.850971 | controller | - ' logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | error: Internal error occurred: 2026-03-09 19:10:03.850975 | controller | error executing command in container: container is not created or running' 2026-03-09 19:10:03.850979 | controller | - ' logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | ++ echo' 2026-03-09 19:10:03.850983 | controller | - ' logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.850988 | controller | - ' logger.go:42: 19:03:33 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-09 19:10:03.850992 | controller | set -euxo pipefail' 2026-03-09 19:10:03.850996 | controller | - ' ' 2026-03-09 19:10:03.851001 | controller | - ' oc project ${NAMESPACE}' 2026-03-09 19:10:03.851005 | controller | - ' # Get pod names for each watcher service' 2026-03-09 19:10:03.851011 | controller | - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.851017 | controller | - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.851023 | controller | - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-09 19:10:03.851029 | controller | - ' ' 2026-03-09 19:10:03.851035 | controller | - ' # Verify memcached mTLS config parameters in watcher-api config' 2026-03-09 19:10:03.851044 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.851049 | controller | - ' echo "Checking watcher-api config..."' 2026-03-09 19:10:03.851055 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851061 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.851066 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851071 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.851077 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851083 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.851087 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851091 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.851096 | controller | - ' ' 2026-03-09 19:10:03.851100 | controller | - ' # Verify mTLS config parameters in memcached backend config' 2026-03-09 19:10:03.851104 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851108 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.851112 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851119 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.851131 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851136 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.851140 | controller | - ' else' 2026-03-09 19:10:03.851156 | controller | - ' exit 1' 2026-03-09 19:10:03.851163 | controller | - ' fi' 2026-03-09 19:10:03.851167 | controller | - ' ' 2026-03-09 19:10:03.851171 | controller | - ' # Verify memcached mTLS config parameters in watcher-applier config' 2026-03-09 19:10:03.851176 | controller | - ' if [ -n "${APPLIERPOD}" ]; then' 2026-03-09 19:10:03.851180 | controller | - ' echo "Checking watcher-applier config..."' 2026-03-09 19:10:03.851184 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851188 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.851193 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851197 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.851201 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851205 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.851210 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851214 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.851220 | controller | - ' ' 2026-03-09 19:10:03.851225 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851229 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.851233 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851238 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.851242 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851246 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.851251 | controller | - ' else' 2026-03-09 19:10:03.851255 | controller | - ' exit 1' 2026-03-09 19:10:03.851259 | controller | - ' fi' 2026-03-09 19:10:03.851263 | controller | - ' ' 2026-03-09 19:10:03.851268 | controller | - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' 2026-03-09 19:10:03.851273 | controller | - ' if [ -n "${DECISIONENGINEPOD}" ]; then' 2026-03-09 19:10:03.851278 | controller | - ' echo "Checking watcher-decision-engine config..."' 2026-03-09 19:10:03.851282 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851286 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-09 19:10:03.851291 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851295 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.851299 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851304 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.851308 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851314 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-09 19:10:03.851318 | controller | - ' ' 2026-03-09 19:10:03.851323 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851327 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-09 19:10:03.851331 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851335 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-09 19:10:03.851340 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.851344 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-09 19:10:03.851348 | controller | - ' else' 2026-03-09 19:10:03.851352 | controller | - ' exit 1' 2026-03-09 19:10:03.851357 | controller | - ' fi' 2026-03-09 19:10:03.851361 | controller | - ' ]' 2026-03-09 19:10:03.851365 | controller | - ' logger.go:42: 19:03:33 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' 2026-03-09 19:10:03.851370 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" 2026-03-09 19:10:03.851374 | controller | on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.851378 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-09 19:10:03.851383 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.851387 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.851400 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier 2026-03-09 19:10:03.851405 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.851409 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' 2026-03-09 19:10:03.851414 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-09 19:10:03.851418 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-09 19:10:03.851423 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.851427 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 2026-03-09 19:10:03.851431 | controller | '']''' 2026-03-09 19:10:03.851436 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api 2026-03-09 19:10:03.851440 | controller | config...''' 2026-03-09 19:10:03.851444 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | Checking watcher-api config...' 2026-03-09 19:10:03.851448 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.851453 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.851457 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.851461 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.851466 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.851470 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.851474 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.851478 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.851483 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.851489 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.851493 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.851497 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.851502 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.851506 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.851510 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.851514 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.851518 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.851523 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.851527 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.851531 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.851535 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.851540 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.851544 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.851548 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.851553 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.851557 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.851561 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.851565 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.851569 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.851574 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.851578 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.851597 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.851603 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.851608 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.851619 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.851624 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.851628 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.851633 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.851637 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile 2026-03-09 19:10:03.851641 | controller | = /etc/pki/tls/private/mtls.key''' 2026-03-09 19:10:03.851645 | controller | - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.851650 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.851656 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.851660 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.851665 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.851669 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.851673 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.851680 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.851685 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.851689 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.851693 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.851698 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.851702 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.851706 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.851712 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.851716 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.851720 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.851725 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.851729 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.851733 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.851737 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.851742 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.851746 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.851750 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.851754 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.851759 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.851763 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.851767 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.851771 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.851776 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.851780 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.851784 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.851788 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.851792 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.851799 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.851803 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.851815 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile 2026-03-09 19:10:03.851819 | controller | = /etc/pki/tls/certs/mtls-ca.crt''' 2026-03-09 19:10:03.851824 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.851828 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.851832 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.851837 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.851841 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.851845 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.851850 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.851854 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.851858 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.851862 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.851867 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.851871 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.851875 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.851879 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.851884 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.851888 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.851892 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.851896 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.851901 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.851905 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.851909 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.851913 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.851917 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.851922 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.851926 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.851930 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.851934 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.851939 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.851946 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.851950 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.851955 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.851960 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.851965 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.851969 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.851973 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.851978 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.851982 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled 2026-03-09 19:10:03.851986 | controller | = true''' 2026-03-09 19:10:03.851990 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.851995 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.851999 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.852012 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.852018 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.852025 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.852031 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.852037 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.852041 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.852046 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852050 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.852054 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.852058 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.852062 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852067 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852071 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.852075 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.852079 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852083 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852090 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.852095 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.852099 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852103 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.852109 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.852114 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.852118 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.852122 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.852127 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.852131 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.852135 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.852139 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.852155 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.852162 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.852166 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.852171 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.852175 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.852179 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' 2026-03-09 19:10:03.852183 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.852188 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.852192 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.852197 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.852203 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.852209 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.852225 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.852232 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.852236 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.852241 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852245 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.852249 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.852253 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.852257 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852262 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852266 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.852270 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.852277 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852281 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852285 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.852290 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.852294 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852300 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.852305 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.852309 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.852313 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.852318 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.852322 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.852326 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.852330 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.852334 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.852339 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.852343 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.852347 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.852351 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.852356 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.852360 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' 2026-03-09 19:10:03.852364 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.852368 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.852373 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.852377 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.852381 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.852386 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.852390 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.852394 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.852398 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.852410 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852414 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.852419 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.852425 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.852430 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852434 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852438 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.852442 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.852447 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852451 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852455 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.852459 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.852463 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852468 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.852472 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.852476 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.852480 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.852485 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.852489 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.852493 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.852497 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.852502 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.852506 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.852510 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.852514 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.852519 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.852523 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.852527 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' 2026-03-09 19:10:03.852531 | controller | - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-09 19:10:03.852536 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.852540 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.852544 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.852548 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.852555 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.852559 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.852565 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.852570 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.852574 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852578 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.852582 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.852588 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.852599 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852604 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852608 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.852613 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.852617 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852621 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852625 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.852630 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.852634 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852638 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.852642 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.852646 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.852651 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.852655 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.852659 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.852663 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.852668 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.852672 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.852676 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.852680 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.852684 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.852689 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.852693 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.852697 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0 2026-03-09 19:10:03.852701 | controller | '']''' 2026-03-09 19:10:03.852706 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier 2026-03-09 19:10:03.852710 | controller | config...''' 2026-03-09 19:10:03.852714 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | Checking watcher-applier 2026-03-09 19:10:03.852720 | controller | config...' 2026-03-09 19:10:03.852724 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.852729 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.852733 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.852737 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.852741 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.852746 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.852750 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.852754 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.852759 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.852763 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.852767 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852771 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.852775 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.852780 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.852784 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852794 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852799 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.852803 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.852807 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852811 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852816 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.852820 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.852824 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852828 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.852833 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.852837 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.852841 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.852847 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.852851 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.852856 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.852860 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.852867 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.852871 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.852875 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.852879 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.852884 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile 2026-03-09 19:10:03.852888 | controller | = /etc/pki/tls/private/mtls.key''' 2026-03-09 19:10:03.852892 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.852896 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.852901 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.852905 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.852909 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.852913 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.852918 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.852922 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.852926 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852930 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.852935 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.852939 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.852943 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852947 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852951 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.852956 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.852960 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.852964 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.852968 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.852973 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.852983 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.852988 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.852994 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.852998 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.853002 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.853008 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.853016 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.853022 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.853028 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.853034 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.853040 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.853044 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.853048 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.853056 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile 2026-03-09 19:10:03.853061 | controller | = /etc/pki/tls/certs/mtls-ca.crt''' 2026-03-09 19:10:03.853065 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.853069 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.853074 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.853078 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.853082 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.853087 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.853091 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.853096 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.853100 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853104 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.853109 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.853113 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.853117 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853121 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853125 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.853130 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.853134 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853140 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853156 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.853163 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.853167 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853172 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.853178 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.853183 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.853187 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.853191 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.853204 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.853208 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.853213 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.853217 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.853221 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.853226 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.853230 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.853234 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled 2026-03-09 19:10:03.853238 | controller | = true''' 2026-03-09 19:10:03.853243 | controller | - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.853247 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.853251 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.853255 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.853260 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.853264 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.853268 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.853272 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.853277 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853281 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.853285 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.853289 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.853294 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853298 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853302 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.853306 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.853311 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853315 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853319 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.853325 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.853329 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853334 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.853338 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.853344 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.853348 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.853352 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.853357 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.853361 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.853365 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.853369 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.853374 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.853378 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.853390 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.853394 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' 2026-03-09 19:10:03.853398 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.853403 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.853407 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.853412 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.853416 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.853420 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.853424 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.853429 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.853433 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853437 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.853441 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.853447 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.853452 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853456 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853460 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.853464 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.853470 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853475 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853480 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.853484 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.853489 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853493 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.853497 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.853502 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.853523 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.853528 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.853532 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.853537 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.853541 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.853545 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.853549 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.853554 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.853558 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.853562 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' 2026-03-09 19:10:03.853566 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.853571 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.853575 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.853579 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.853589 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.853593 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.853598 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.853602 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.853606 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853610 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.853615 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.853619 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.853623 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853630 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853634 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.853638 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.853643 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853647 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853651 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.853655 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.853659 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853664 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.853668 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.853672 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.853676 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.853681 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.853685 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.853689 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.853693 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.853698 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.853702 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.853708 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.853713 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.853717 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' 2026-03-09 19:10:03.853722 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-09 19:10:03.853726 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.853730 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.853734 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.853739 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.853743 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.853747 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.853752 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.853756 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853766 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.853772 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.853777 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.853781 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853785 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853789 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.853794 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.853798 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853802 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853806 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.853811 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.853815 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853819 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.853823 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.853828 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-09 19:10:03.853832 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.853836 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.853840 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.853845 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.853849 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.853853 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.853857 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.853872 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.853876 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.853881 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0 2026-03-09 19:10:03.853885 | controller | '']''' 2026-03-09 19:10:03.853889 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine 2026-03-09 19:10:03.853893 | controller | config...''' 2026-03-09 19:10:03.853898 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine 2026-03-09 19:10:03.853902 | controller | config...' 2026-03-09 19:10:03.853906 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-09 19:10:03.853910 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-09 19:10:03.853915 | controller | - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.853919 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.853923 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.853928 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.853934 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.853938 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.853944 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.853949 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.853953 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.853957 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.853961 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.853973 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.853977 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853982 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.853986 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.853990 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.853995 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.853999 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854003 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.854009 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.854015 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854021 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.854027 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.854033 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.854041 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.854045 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.854050 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.854054 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.854058 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.854062 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.854067 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.854071 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.854075 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.854079 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.854084 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.854088 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile 2026-03-09 19:10:03.854096 | controller | = /etc/pki/tls/private/mtls.key''' 2026-03-09 19:10:03.854100 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.854104 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.854109 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.854113 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.854117 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.854122 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.854126 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.854130 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.854135 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854139 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.854143 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.854169 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.854173 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854178 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854182 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.854186 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.854199 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854203 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854208 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.854212 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.854216 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854221 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.854225 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.854229 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.854234 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.854238 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.854242 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.854247 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.854251 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.854255 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.854261 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.854266 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.854270 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.854274 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.854279 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.854283 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile 2026-03-09 19:10:03.854287 | controller | = /etc/pki/tls/certs/mtls-ca.crt''' 2026-03-09 19:10:03.854291 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.854296 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.854300 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.854304 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.854308 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.854313 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.854317 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.854321 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.854326 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854330 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.854337 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.854341 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.854346 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854350 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854354 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.854358 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.854363 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854367 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854371 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.854376 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.854386 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854391 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.854397 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.854401 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.854407 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.854413 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.854418 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.854422 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.854426 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.854431 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.854435 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.854439 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.854443 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.854448 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.854452 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.854456 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled 2026-03-09 19:10:03.854461 | controller | = true''' 2026-03-09 19:10:03.854465 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.854469 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.854473 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.854478 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.854482 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.854486 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.854491 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.854495 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.854499 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854504 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.854508 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.854512 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.854516 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854521 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854525 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.854529 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.854533 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854538 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854542 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.854546 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.854552 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854557 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.854561 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.854565 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.854570 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.854581 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.854587 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.854591 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.854596 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.854600 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.854604 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.854609 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.854613 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.854617 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.854622 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.854626 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' 2026-03-09 19:10:03.854630 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.854634 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.854639 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.854643 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.854647 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.854653 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.854658 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.854662 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.854666 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854671 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.854675 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.854679 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.854683 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854688 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854692 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.854698 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.854702 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854706 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854711 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.854715 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.854719 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854723 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.854728 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.854732 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.854736 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.854741 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.854745 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.854749 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.854753 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.854764 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.854769 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.854773 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.854777 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.854782 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.854786 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.854790 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' 2026-03-09 19:10:03.854794 | controller | - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.854799 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.854803 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.854807 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.854811 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.854816 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.854820 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.854824 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.854828 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854833 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.854839 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.854843 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.854849 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854854 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854859 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.854863 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.854868 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.854872 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.854876 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.854881 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.854885 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.854889 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.854893 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.854898 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.854902 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.854906 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.854910 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.854915 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.854919 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.854923 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.854928 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.854932 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.854936 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.854940 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.854951 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.854956 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' 2026-03-09 19:10:03.854960 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-09 19:10:03.854965 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.854969 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-09 19:10:03.854973 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.854978 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-09 19:10:03.854982 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.854988 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.854992 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.854996 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.855001 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.855005 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.855011 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.855017 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.855023 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.855029 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.855035 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.855041 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.855047 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.855052 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.855058 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.855063 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.855069 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.855074 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.855080 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.855085 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.855090 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-09 19:10:03.855095 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-09 19:10:03.855099 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.855106 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-09 19:10:03.855110 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.855114 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.855119 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.855123 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.855127 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.855132 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.855136 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | test step completed 2-patch-mtls' 2026-03-09 19:10:03.855140 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | starting test 2026-03-09 19:10:03.855157 | controller | step 3-disable-podlevel-tls' 2026-03-09 19:10:03.855164 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-09 19:10:03.855180 | controller | [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": 2026-03-09 19:10:03.855185 | controller | "replace", "path": "/spec/apiServiceTemplate/tls/api", "value":{ "internal": {}, 2026-03-09 19:10:03.855189 | controller | "public": {} }}]''' 2026-03-09 19:10:03.855194 | controller | - ' ]' 2026-03-09 19:10:03.855198 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.855202 | controller | patched' 2026-03-09 19:10:03.855207 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-09 19:10:03.855211 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.855215 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.855220 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.855224 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.855228 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.855232 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.855237 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.855241 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.855245 | controller | ]' 2026-03-09 19:10:03.855252 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-09 19:10:03.855256 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.855260 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-09 19:10:03.855265 | controller | - ' # check that watcher public endpoint does use https' 2026-03-09 19:10:03.855269 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.855273 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-09 19:10:03.855277 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.855282 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.855286 | controller | == "" ]; then' 2026-03-09 19:10:03.855290 | controller | - ' exit 0' 2026-03-09 19:10:03.855295 | controller | - ' fi' 2026-03-09 19:10:03.855299 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.855303 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.855307 | controller | - ' counter=0' 2026-03-09 19:10:03.855312 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.855316 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.855320 | controller | - ' echo ${i}' 2026-03-09 19:10:03.855325 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.855329 | controller | - ' fi' 2026-03-09 19:10:03.855333 | controller | - ' done' 2026-03-09 19:10:03.855337 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.855342 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.855346 | controller | - ' exit 1' 2026-03-09 19:10:03.855350 | controller | - ' else' 2026-03-09 19:10:03.855355 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.855359 | controller | - ' fi' 2026-03-09 19:10:03.855363 | controller | - ' ]' 2026-03-09 19:10:03.855367 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.855371 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.855376 | controller | -c Type' 2026-03-09 19:10:03.855382 | controller | - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.855389 | controller | - ' logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.855393 | controller | 1 '']''' 2026-03-09 19:10:03.855397 | controller | - ' logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-09 19:10:03.855402 | controller | - ' logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-09 19:10:03.855406 | controller | $1}''' 2026-03-09 19:10:03.855410 | controller | - ' logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-09 19:10:03.855414 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.855419 | controller | -c Type -c ID' 2026-03-09 19:10:03.855423 | controller | - ' logger.go:42: 19:03:43 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.855427 | controller | - ' logger.go:42: 19:03:43 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.855431 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.855436 | controller | - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.855440 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.855444 | controller | - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.855448 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.855453 | controller | - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.855457 | controller | '']''' 2026-03-09 19:10:03.855461 | controller | - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.855472 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.855476 | controller | - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-09 19:10:03.855481 | controller | - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-09 19:10:03.855485 | controller | - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-09 19:10:03.855489 | controller | - ' logger.go:42: 19:03:46 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.855493 | controller | 0 '']''' 2026-03-09 19:10:03.855501 | controller | - ' logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-09 19:10:03.855505 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.855509 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.855515 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.855520 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.855524 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.855528 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.855532 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.855537 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.855541 | controller | ]' 2026-03-09 19:10:03.855545 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-09 19:10:03.855550 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.855554 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-09 19:10:03.855558 | controller | - ' # check that watcher public endpoint does use https' 2026-03-09 19:10:03.855562 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.855568 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-09 19:10:03.855573 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.855593 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.855602 | controller | == "" ]; then' 2026-03-09 19:10:03.855607 | controller | - ' exit 0' 2026-03-09 19:10:03.855611 | controller | - ' fi' 2026-03-09 19:10:03.855615 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.855619 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.855623 | controller | - ' counter=0' 2026-03-09 19:10:03.855628 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.855632 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.855636 | controller | - ' echo ${i}' 2026-03-09 19:10:03.855641 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.855645 | controller | - ' fi' 2026-03-09 19:10:03.855649 | controller | - ' done' 2026-03-09 19:10:03.855653 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.855658 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.855662 | controller | - ' exit 1' 2026-03-09 19:10:03.855666 | controller | - ' else' 2026-03-09 19:10:03.855670 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.855674 | controller | - ' fi' 2026-03-09 19:10:03.855679 | controller | - ' ]' 2026-03-09 19:10:03.855683 | controller | - ' logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.855687 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.855691 | controller | -c Type' 2026-03-09 19:10:03.855696 | controller | - ' logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.855700 | controller | - ' logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.855704 | controller | 1 '']''' 2026-03-09 19:10:03.855708 | controller | - ' logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-09 19:10:03.855713 | controller | - ' logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-09 19:10:03.855717 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.855721 | controller | -c Type -c ID' 2026-03-09 19:10:03.855725 | controller | - ' logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-09 19:10:03.855729 | controller | $1}''' 2026-03-09 19:10:03.855734 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.855738 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.855742 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.855746 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.855751 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.855755 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.855759 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.855770 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.855775 | controller | '']''' 2026-03-09 19:10:03.855779 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-09 19:10:03.855783 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-09 19:10:03.855790 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.855794 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.855798 | controller | - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-09 19:10:03.855803 | controller | - ' logger.go:42: 19:03:53 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.855807 | controller | 0 '']''' 2026-03-09 19:10:03.855811 | controller | - ' logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-09 19:10:03.855815 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.855820 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.855824 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.855828 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.855832 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.855837 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.855841 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.855845 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.855849 | controller | ]' 2026-03-09 19:10:03.855854 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-09 19:10:03.855858 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.855862 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-09 19:10:03.855867 | controller | - ' # check that watcher public endpoint does use https' 2026-03-09 19:10:03.855871 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.855875 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-09 19:10:03.855879 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.855884 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.855888 | controller | == "" ]; then' 2026-03-09 19:10:03.855892 | controller | - ' exit 0' 2026-03-09 19:10:03.855896 | controller | - ' fi' 2026-03-09 19:10:03.855901 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.855905 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.855909 | controller | - ' counter=0' 2026-03-09 19:10:03.855914 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.855918 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.855922 | controller | - ' echo ${i}' 2026-03-09 19:10:03.855927 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.855931 | controller | - ' fi' 2026-03-09 19:10:03.855935 | controller | - ' done' 2026-03-09 19:10:03.855939 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.855947 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.855951 | controller | - ' exit 1' 2026-03-09 19:10:03.855956 | controller | - ' else' 2026-03-09 19:10:03.855960 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.855964 | controller | - ' fi' 2026-03-09 19:10:03.855969 | controller | - ' ]' 2026-03-09 19:10:03.855975 | controller | - ' logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.855979 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.855983 | controller | -c Type' 2026-03-09 19:10:03.855988 | controller | - ' logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.855994 | controller | - ' logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.855998 | controller | 1 '']''' 2026-03-09 19:10:03.856002 | controller | - ' logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-09 19:10:03.856009 | controller | - ' logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-09 19:10:03.856015 | controller | $1}''' 2026-03-09 19:10:03.856022 | controller | - ' logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-09 19:10:03.856028 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.856034 | controller | -c Type -c ID' 2026-03-09 19:10:03.856039 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.856045 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.856050 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.856056 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.856062 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.856068 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.856074 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.856089 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.856095 | controller | '']''' 2026-03-09 19:10:03.856100 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.856106 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.856112 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-09 19:10:03.856117 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-09 19:10:03.856122 | controller | - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-09 19:10:03.856128 | controller | - ' logger.go:42: 19:04:00 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.856133 | controller | 0 '']''' 2026-03-09 19:10:03.856137 | controller | - ' logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-09 19:10:03.856142 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.856161 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.856166 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.856170 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.856174 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.856181 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.856186 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.856190 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.856194 | controller | ]' 2026-03-09 19:10:03.856199 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-09 19:10:03.856203 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.856207 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-09 19:10:03.856212 | controller | - ' # check that watcher public endpoint does use https' 2026-03-09 19:10:03.856216 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.856222 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-09 19:10:03.856227 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.856231 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.856235 | controller | == "" ]; then' 2026-03-09 19:10:03.856240 | controller | - ' exit 0' 2026-03-09 19:10:03.856244 | controller | - ' fi' 2026-03-09 19:10:03.856248 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.856253 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.856257 | controller | - ' counter=0' 2026-03-09 19:10:03.856261 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.856266 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.856270 | controller | - ' echo ${i}' 2026-03-09 19:10:03.856274 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.856281 | controller | - ' fi' 2026-03-09 19:10:03.856286 | controller | - ' done' 2026-03-09 19:10:03.856290 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.856294 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.856299 | controller | - ' exit 1' 2026-03-09 19:10:03.856303 | controller | - ' else' 2026-03-09 19:10:03.856307 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.856311 | controller | - ' fi' 2026-03-09 19:10:03.856316 | controller | - ' ]' 2026-03-09 19:10:03.856320 | controller | - ' logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.856324 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.856328 | controller | -c Type' 2026-03-09 19:10:03.856333 | controller | - ' logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.856337 | controller | - ' logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.856341 | controller | 1 '']''' 2026-03-09 19:10:03.856346 | controller | - ' logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-09 19:10:03.856350 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.856354 | controller | -c Type -c ID' 2026-03-09 19:10:03.856358 | controller | - ' logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-09 19:10:03.856363 | controller | $1}''' 2026-03-09 19:10:03.856367 | controller | - ' logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-09 19:10:03.856371 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.856375 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.856379 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.856384 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.856388 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.856400 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.856404 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.856409 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.856413 | controller | '']''' 2026-03-09 19:10:03.856418 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-09 19:10:03.856423 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-09 19:10:03.856431 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-09 19:10:03.856435 | controller | - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.856439 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.856443 | controller | - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 == 2026-03-09 19:10:03.856448 | controller | 0 '']''' 2026-03-09 19:10:03.856452 | controller | - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + grep public' 2026-03-09 19:10:03.856456 | controller | - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-09 19:10:03.856460 | controller | - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-09 19:10:03.856465 | controller | - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.856469 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.856473 | controller | - ' logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.856477 | controller | 1 '']''' 2026-03-09 19:10:03.856482 | controller | - ' logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods 2026-03-09 19:10:03.856486 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.856490 | controller | - ' logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + ''['' '''' 2026-03-09 19:10:03.856495 | controller | == '''' '']''' 2026-03-09 19:10:03.856499 | controller | - ' logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + exit 0' 2026-03-09 19:10:03.856503 | controller | - ' logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-09 19:10:03.856507 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.856512 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.856516 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.856520 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.856524 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.856529 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.856533 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.856537 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.856541 | controller | ]' 2026-03-09 19:10:03.856546 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-09 19:10:03.856550 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.856554 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-09 19:10:03.856558 | controller | - ' # check that watcher public endpoint does use https' 2026-03-09 19:10:03.856563 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.856567 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-09 19:10:03.856571 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.856575 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.856580 | controller | == "" ]; then' 2026-03-09 19:10:03.856584 | controller | - ' exit 0' 2026-03-09 19:10:03.856588 | controller | - ' fi' 2026-03-09 19:10:03.856592 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.856597 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.856601 | controller | - ' counter=0' 2026-03-09 19:10:03.856605 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.856611 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.856615 | controller | - ' echo ${i}' 2026-03-09 19:10:03.856620 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.856624 | controller | - ' fi' 2026-03-09 19:10:03.856628 | controller | - ' done' 2026-03-09 19:10:03.856633 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.856637 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.856641 | controller | - ' exit 1' 2026-03-09 19:10:03.856645 | controller | - ' else' 2026-03-09 19:10:03.856650 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.856654 | controller | - ' fi' 2026-03-09 19:10:03.856658 | controller | - ' ]' 2026-03-09 19:10:03.856663 | controller | - ' logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.856667 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.856671 | controller | -c Type' 2026-03-09 19:10:03.856682 | controller | - ' logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.856687 | controller | - ' logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.856692 | controller | 1 '']''' 2026-03-09 19:10:03.856696 | controller | - ' logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-09 19:10:03.856700 | controller | - ' logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-09 19:10:03.856704 | controller | $1}''' 2026-03-09 19:10:03.856709 | controller | - ' logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-09 19:10:03.856713 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.856717 | controller | -c Type -c ID' 2026-03-09 19:10:03.856721 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.856726 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.856730 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.856734 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.856738 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.856743 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-09 19:10:03.856747 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.856751 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.856755 | controller | '']''' 2026-03-09 19:10:03.856760 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-09 19:10:03.856764 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-09 19:10:03.856770 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.856774 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.856778 | controller | - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-09 19:10:03.856782 | controller | - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 == 2026-03-09 19:10:03.856787 | controller | 0 '']''' 2026-03-09 19:10:03.856791 | controller | - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + grep public' 2026-03-09 19:10:03.856795 | controller | - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-09 19:10:03.856799 | controller | - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-09 19:10:03.856806 | controller | - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-09 19:10:03.856810 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-09 19:10:03.856814 | controller | - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-09 19:10:03.856818 | controller | 1 '']''' 2026-03-09 19:10:03.856823 | controller | - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods 2026-03-09 19:10:03.856827 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.856831 | controller | - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + ''['' '''' 2026-03-09 19:10:03.856836 | controller | == '''' '']''' 2026-03-09 19:10:03.856840 | controller | - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + exit 0' 2026-03-09 19:10:03.856844 | controller | - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | test step completed 2026-03-09 19:10:03.856848 | controller | 3-disable-podlevel-tls' 2026-03-09 19:10:03.856853 | controller | - ' logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | starting test 2026-03-09 19:10:03.856857 | controller | step 4-deploy-without-route' 2026-03-09 19:10:03.856861 | controller | - ' logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | running command: 2026-03-09 19:10:03.856865 | controller | [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": 2026-03-09 19:10:03.856870 | controller | "replace", "path": "/spec/apiServiceTemplate/override", "value":{"service": { "internal": 2026-03-09 19:10:03.856874 | controller | {}, "public": { "metadata": { "annotations": { "metallb.universe.tf/address-pool": 2026-03-09 19:10:03.856878 | controller | "ctlplane", "metallb.universe.tf/allow-shared-ip": "ctlplane" } }, "spec": { "type": 2026-03-09 19:10:03.856884 | controller | "LoadBalancer" } } } }}]''' 2026-03-09 19:10:03.856890 | controller | - ' ]' 2026-03-09 19:10:03.856897 | controller | - ' logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.856902 | controller | patched' 2026-03-09 19:10:03.856909 | controller | - ' logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | running command: 2026-03-09 19:10:03.856914 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.856919 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.856923 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.856927 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.856939 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.856944 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.856948 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.856955 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.856960 | controller | ]' 2026-03-09 19:10:03.856964 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.856968 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.856973 | controller | == "" ]; then' 2026-03-09 19:10:03.856977 | controller | - ' exit 0' 2026-03-09 19:10:03.856981 | controller | - ' fi' 2026-03-09 19:10:03.856986 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.856990 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.856994 | controller | - ' counter=0' 2026-03-09 19:10:03.856998 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.857003 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.857008 | controller | - ' echo ${i}' 2026-03-09 19:10:03.857017 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.857023 | controller | - ' fi' 2026-03-09 19:10:03.857029 | controller | - ' done' 2026-03-09 19:10:03.857035 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.857041 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.857047 | controller | - ' exit 1' 2026-03-09 19:10:03.857053 | controller | - ' else' 2026-03-09 19:10:03.857058 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.857064 | controller | - ' fi' 2026-03-09 19:10:03.857069 | controller | - ' ]' 2026-03-09 19:10:03.857075 | controller | - ' logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.857081 | controller | - ' logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | + oc exec -n 2026-03-09 19:10:03.857086 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.857090 | controller | -c Type' 2026-03-09 19:10:03.857095 | controller | - ' logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | + ''['' 1 == 2026-03-09 19:10:03.857099 | controller | 1 '']''' 2026-03-09 19:10:03.857103 | controller | - ' logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ oc exec -n 2026-03-09 19:10:03.857107 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.857112 | controller | -c Type -c ID' 2026-03-09 19:10:03.857116 | controller | - ' logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ awk ''{print 2026-03-09 19:10:03.857120 | controller | $1}''' 2026-03-09 19:10:03.857124 | controller | - ' logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ grep watcher' 2026-03-09 19:10:03.857129 | controller | - ' logger.go:42: 19:04:24 | watcher-tls/4-deploy-without-route | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.857133 | controller | - ' logger.go:42: 19:04:24 | watcher-tls/4-deploy-without-route | ++ oc get -n 2026-03-09 19:10:03.857137 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.857141 | controller | - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.857165 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.857173 | controller | - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | ++ oc get -n 2026-03-09 19:10:03.857180 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.857185 | controller | - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.857189 | controller | '']''' 2026-03-09 19:10:03.857194 | controller | - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | ++ oc get pods 2026-03-09 19:10:03.857198 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.857202 | controller | - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + ''['' '''' 2026-03-09 19:10:03.857206 | controller | == '''' '']''' 2026-03-09 19:10:03.857211 | controller | - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + exit 0' 2026-03-09 19:10:03.857215 | controller | - ' logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | running command: 2026-03-09 19:10:03.857219 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.857223 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.857228 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.857232 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.857236 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.857241 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.857247 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.857251 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.857256 | controller | ]' 2026-03-09 19:10:03.857261 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.857267 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.857273 | controller | == "" ]; then' 2026-03-09 19:10:03.857290 | controller | - ' exit 0' 2026-03-09 19:10:03.857296 | controller | - ' fi' 2026-03-09 19:10:03.857300 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.857305 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.857309 | controller | - ' counter=0' 2026-03-09 19:10:03.857313 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.857317 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.857322 | controller | - ' echo ${i}' 2026-03-09 19:10:03.857326 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.857330 | controller | - ' fi' 2026-03-09 19:10:03.857335 | controller | - ' done' 2026-03-09 19:10:03.857339 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.857343 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.857347 | controller | - ' exit 1' 2026-03-09 19:10:03.857352 | controller | - ' else' 2026-03-09 19:10:03.857356 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.857360 | controller | - ' fi' 2026-03-09 19:10:03.857365 | controller | - ' ]' 2026-03-09 19:10:03.857369 | controller | - ' logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | + oc exec -n 2026-03-09 19:10:03.857373 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.857377 | controller | -c Type' 2026-03-09 19:10:03.857382 | controller | - ' logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.857386 | controller | - ' logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | + ''['' 1 == 2026-03-09 19:10:03.857390 | controller | 1 '']''' 2026-03-09 19:10:03.857395 | controller | - ' logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ oc exec -n 2026-03-09 19:10:03.857399 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.857403 | controller | -c Type -c ID' 2026-03-09 19:10:03.857407 | controller | - ' logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ awk ''{print 2026-03-09 19:10:03.857412 | controller | $1}''' 2026-03-09 19:10:03.857416 | controller | - ' logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ grep watcher' 2026-03-09 19:10:03.857420 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.857425 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get -n 2026-03-09 19:10:03.857429 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.857433 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.857437 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.857442 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get -n 2026-03-09 19:10:03.857446 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.857450 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.857455 | controller | '']''' 2026-03-09 19:10:03.857459 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get pods 2026-03-09 19:10:03.857465 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.857470 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + ''['' '''' 2026-03-09 19:10:03.857474 | controller | == '''' '']''' 2026-03-09 19:10:03.857478 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + exit 0' 2026-03-09 19:10:03.857483 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | test step completed 2026-03-09 19:10:03.857487 | controller | 4-deploy-without-route' 2026-03-09 19:10:03.857491 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | starting test step 5-disable-tls' 2026-03-09 19:10:03.857495 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | running command: [sh -c 2026-03-09 19:10:03.857500 | controller | oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace", 2026-03-09 19:10:03.857504 | controller | "path": "/spec/apiServiceTemplate/override", "value":{}}]''' 2026-03-09 19:10:03.857508 | controller | - ' ]' 2026-03-09 19:10:03.857513 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.857517 | controller | patched' 2026-03-09 19:10:03.857521 | controller | - ' logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | running command: [sh -c 2026-03-09 19:10:03.857525 | controller | set -euxo pipefail' 2026-03-09 19:10:03.857530 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.857534 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.857538 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.857542 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.857547 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.857551 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.857555 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.857559 | controller | ]' 2026-03-09 19:10:03.857564 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.857575 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.857580 | controller | == "" ]; then' 2026-03-09 19:10:03.857584 | controller | - ' exit 0' 2026-03-09 19:10:03.857588 | controller | - ' fi' 2026-03-09 19:10:03.857593 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.857597 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.857601 | controller | - ' counter=0' 2026-03-09 19:10:03.857608 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.857612 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.857617 | controller | - ' echo ${i}' 2026-03-09 19:10:03.857621 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.857627 | controller | - ' fi' 2026-03-09 19:10:03.857631 | controller | - ' done' 2026-03-09 19:10:03.857636 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.857640 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.857644 | controller | - ' exit 1' 2026-03-09 19:10:03.857648 | controller | - ' else' 2026-03-09 19:10:03.857653 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.857657 | controller | - ' fi' 2026-03-09 19:10:03.857661 | controller | - ' # check that no watcher endpoint uses https' 2026-03-09 19:10:03.857666 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.857670 | controller | infra-optim | [ $(grep -c https) == 0 ]' 2026-03-09 19:10:03.857676 | controller | - ' ]' 2026-03-09 19:10:03.857680 | controller | - ' logger.go:42: 19:04:31 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.857685 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.857689 | controller | - ' logger.go:42: 19:04:31 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.857695 | controller | - ' logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.857699 | controller | - ' logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ grep watcher' 2026-03-09 19:10:03.857703 | controller | - ' logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ awk ''{print $1}''' 2026-03-09 19:10:03.857708 | controller | - ' logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.857712 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.857716 | controller | - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.857720 | controller | - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.857725 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.857729 | controller | - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.857733 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.857737 | controller | - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.857742 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.857748 | controller | - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.857752 | controller | '']''' 2026-03-09 19:10:03.857757 | controller | - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators 2026-03-09 19:10:03.857761 | controller | -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.857765 | controller | - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']''' 2026-03-09 19:10:03.857769 | controller | - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + exit 0' 2026-03-09 19:10:03.857774 | controller | - ' logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | running command: [sh -c 2026-03-09 19:10:03.857778 | controller | set -euxo pipefail' 2026-03-09 19:10:03.857782 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-09 19:10:03.857787 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.857791 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-09 19:10:03.857795 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.857799 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.857804 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.857808 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.857812 | controller | ]' 2026-03-09 19:10:03.857817 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.857821 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.857825 | controller | == "" ]; then' 2026-03-09 19:10:03.857829 | controller | - ' exit 0' 2026-03-09 19:10:03.857834 | controller | - ' fi' 2026-03-09 19:10:03.857838 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.857842 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.857847 | controller | - ' counter=0' 2026-03-09 19:10:03.857853 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.857864 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.857869 | controller | - ' echo ${i}' 2026-03-09 19:10:03.857873 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.857877 | controller | - ' fi' 2026-03-09 19:10:03.857882 | controller | - ' done' 2026-03-09 19:10:03.857886 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.857890 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.857895 | controller | - ' exit 1' 2026-03-09 19:10:03.857899 | controller | - ' else' 2026-03-09 19:10:03.857903 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.857907 | controller | - ' fi' 2026-03-09 19:10:03.857912 | controller | - ' # check that no watcher endpoint uses https' 2026-03-09 19:10:03.857916 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-09 19:10:03.857920 | controller | infra-optim | [ $(grep -c https) == 0 ]' 2026-03-09 19:10:03.857925 | controller | - ' ]' 2026-03-09 19:10:03.857929 | controller | - ' logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.857933 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.857937 | controller | - ' logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.857942 | controller | - ' logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.857946 | controller | - ' logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ grep watcher' 2026-03-09 19:10:03.857950 | controller | - ' logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ awk ''{print $1}''' 2026-03-09 19:10:03.857958 | controller | - ' logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.857962 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.857966 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' 2026-03-09 19:10:03.857971 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.857975 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.857979 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 2026-03-09 19:10:03.857983 | controller | == 1881642dbc2749d28b8c91d8865efef3 '']''' 2026-03-09 19:10:03.857988 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.857992 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.858001 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.858007 | controller | '']''' 2026-03-09 19:10:03.858012 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators 2026-03-09 19:10:03.858019 | controller | -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.858025 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']''' 2026-03-09 19:10:03.858031 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + exit 0' 2026-03-09 19:10:03.858037 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | test step completed 5-disable-tls' 2026-03-09 19:10:03.858043 | controller | - ' logger.go:42: 19:04:40 | watcher-tls/6-cleanup-watcher | starting test step 2026-03-09 19:10:03.858049 | controller | 6-cleanup-watcher' 2026-03-09 19:10:03.858055 | controller | - ' logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | running command: [sh 2026-03-09 19:10:03.858060 | controller | -c set -ex' 2026-03-09 19:10:03.858068 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.858074 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.858079 | controller | - ' ]' 2026-03-09 19:10:03.858083 | controller | - ' logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.858087 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.858091 | controller | - ' logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.858096 | controller | - ' logger.go:42: 19:04:43 | watcher-tls/6-cleanup-watcher | + ''['' 1 == 0 '']''' 2026-03-09 19:10:03.858100 | controller | - ' logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | running command: [sh 2026-03-09 19:10:03.858104 | controller | -c set -ex' 2026-03-09 19:10:03.858108 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.858113 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.858117 | controller | - ' ]' 2026-03-09 19:10:03.858121 | controller | - ' logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.858126 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.858130 | controller | - ' logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.858134 | controller | - ' logger.go:42: 19:04:47 | watcher-tls/6-cleanup-watcher | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.858138 | controller | - ' logger.go:42: 19:04:47 | watcher-tls/6-cleanup-watcher | test step completed 2026-03-09 19:10:03.858143 | controller | 6-cleanup-watcher' 2026-03-09 19:10:03.858176 | controller | - ' logger.go:42: 19:04:47 | watcher-tls/7-cleanup-certs | starting test step 7-cleanup-certs' 2026-03-09 19:10:03.858182 | controller | - ' logger.go:42: 19:04:47 | watcher-tls/7-cleanup-certs | test step completed 2026-03-09 19:10:03.858186 | controller | 7-cleanup-certs' 2026-03-09 19:10:03.858190 | controller | - ' logger.go:42: 19:04:47 | watcher-tls | skipping kubernetes event logging' 2026-03-09 19:10:03.858195 | controller | - === CONT kuttl/harness/watcher-rmquser 2026-03-09 19:10:03.858199 | controller | - ' logger.go:42: 19:04:47 | watcher-rmquser | Skipping creation of user-supplied 2026-03-09 19:10:03.858203 | controller | namespace: watcher-kuttl-default' 2026-03-09 19:10:03.858208 | controller | - ' logger.go:42: 19:04:47 | watcher-rmquser/0-cleanup-watcher | starting test 2026-03-09 19:10:03.858212 | controller | step 0-cleanup-watcher' 2026-03-09 19:10:03.858216 | controller | - ' logger.go:42: 19:04:47 | watcher-rmquser/0-cleanup-watcher | test step completed 2026-03-09 19:10:03.858220 | controller | 0-cleanup-watcher' 2026-03-09 19:10:03.858227 | controller | - ' logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | starting test step 1-deploy' 2026-03-09 19:10:03.858231 | controller | - ' logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.858236 | controller | created' 2026-03-09 19:10:03.858240 | controller | - ' logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | running command: [sh -c 2026-03-09 19:10:03.858244 | controller | set -euxo pipefail' 2026-03-09 19:10:03.858249 | controller | - ' ' 2026-03-09 19:10:03.858253 | controller | - ' # Wait for Watcher to be Ready' 2026-03-09 19:10:03.858257 | controller | - ' kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE 2026-03-09 19:10:03.858261 | controller | --timeout=300s' 2026-03-09 19:10:03.858266 | controller | - ' ' 2026-03-09 19:10:03.858270 | controller | - ' # Verify WatcherNotificationTransportURLReady condition exists and is True' 2026-03-09 19:10:03.858274 | controller | - ' kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'' 2026-03-09 19:10:03.858279 | controller | | grep -q "True"' 2026-03-09 19:10:03.858283 | controller | - ' echo "WatcherNotificationTransportURLReady condition is True"' 2026-03-09 19:10:03.858289 | controller | - ' ' 2026-03-09 19:10:03.858293 | controller | - ' # Count TransportURL CRs - should be exactly 2 (one for messaging, one 2026-03-09 19:10:03.858298 | controller | for notifications)' 2026-03-09 19:10:03.858302 | controller | - ' transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep 2026-03-09 19:10:03.858306 | controller | "watcher-kuttl-watcher-transport" | wc -l)' 2026-03-09 19:10:03.858310 | controller | - ' notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o 2026-03-09 19:10:03.858315 | controller | name | grep "watcher-kuttl-watcher-notification" | wc -l)' 2026-03-09 19:10:03.858319 | controller | - ' ' 2026-03-09 19:10:03.858323 | controller | - ' if [ "$transport_count" -ne "1" ]; then' 2026-03-09 19:10:03.858328 | controller | - ' echo "Expected 1 watcher-transport TransportURL, found $transport_count"' 2026-03-09 19:10:03.858332 | controller | - ' exit 1' 2026-03-09 19:10:03.858336 | controller | - ' fi' 2026-03-09 19:10:03.858340 | controller | - ' ' 2026-03-09 19:10:03.858345 | controller | - ' if [ "$notification_transport_count" -ne "1" ]; then' 2026-03-09 19:10:03.858349 | controller | - ' echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"' 2026-03-09 19:10:03.858353 | controller | - ' exit 1' 2026-03-09 19:10:03.858358 | controller | - ' fi' 2026-03-09 19:10:03.858362 | controller | - ' ' 2026-03-09 19:10:03.858366 | controller | - ' echo "Correctly found 2 TransportURLs (separate clusters: transport and 2026-03-09 19:10:03.858370 | controller | notification)"' 2026-03-09 19:10:03.858374 | controller | - ' ' 2026-03-09 19:10:03.858379 | controller | - ' # Verify watcher-transport has correct user and vhost' 2026-03-09 19:10:03.858383 | controller | - ' transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-09 19:10:03.858387 | controller | -n $NAMESPACE -o jsonpath=''{.spec.username}'')' 2026-03-09 19:10:03.858392 | controller | - ' transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-09 19:10:03.858396 | controller | -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' 2026-03-09 19:10:03.858400 | controller | - ' if [ "$transport_user" != "watcher-rpc" ]; then' 2026-03-09 19:10:03.858404 | controller | - ' echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"' 2026-03-09 19:10:03.858409 | controller | - ' exit 1' 2026-03-09 19:10:03.858413 | controller | - ' fi' 2026-03-09 19:10:03.858417 | controller | - ' if [ "$transport_vhost" != "watcher-rpc" ]; then' 2026-03-09 19:10:03.858422 | controller | - ' echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"' 2026-03-09 19:10:03.858426 | controller | - ' exit 1' 2026-03-09 19:10:03.858430 | controller | - ' fi' 2026-03-09 19:10:03.858434 | controller | - ' echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"' 2026-03-09 19:10:03.858438 | controller | - ' ' 2026-03-09 19:10:03.858443 | controller | - ' # Verify notification-transport has correct user and vhost' 2026-03-09 19:10:03.858447 | controller | - ' notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-09 19:10:03.858451 | controller | -n $NAMESPACE -o jsonpath=''{.spec.username}'')' 2026-03-09 19:10:03.858456 | controller | - ' notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-09 19:10:03.858460 | controller | -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' 2026-03-09 19:10:03.858464 | controller | - ' if [ "$notif_user" != "watcher-notifications" ]; then' 2026-03-09 19:10:03.858468 | controller | - ' echo "Expected notification-transport username ''watcher-notifications'', 2026-03-09 19:10:03.858473 | controller | found ''$notif_user''"' 2026-03-09 19:10:03.858477 | controller | - ' exit 1' 2026-03-09 19:10:03.858481 | controller | - ' fi' 2026-03-09 19:10:03.858493 | controller | - ' if [ "$notif_vhost" != "watcher-notifications" ]; then' 2026-03-09 19:10:03.858497 | controller | - ' echo "Expected notification-transport vhost ''watcher-notifications'', 2026-03-09 19:10:03.858503 | controller | found ''$notif_vhost''"' 2026-03-09 19:10:03.858508 | controller | - ' exit 1' 2026-03-09 19:10:03.858512 | controller | - ' fi' 2026-03-09 19:10:03.858516 | controller | - ' echo "Notification transport has correct user (watcher-notifications) and 2026-03-09 19:10:03.858521 | controller | vhost (watcher-notifications)"' 2026-03-09 19:10:03.858525 | controller | - ' ' 2026-03-09 19:10:03.858529 | controller | - ' # Verify that watcher.conf contains the notifications transport_url' 2026-03-09 19:10:03.858533 | controller | - ' WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api" 2026-03-09 19:10:03.858538 | controller | -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)' 2026-03-09 19:10:03.858542 | controller | - ' if [ -z "${WATCHER_API_POD}" ]; then' 2026-03-09 19:10:03.858546 | controller | - ' echo "No watcher-api pod found"' 2026-03-09 19:10:03.858551 | controller | - ' exit 1' 2026-03-09 19:10:03.858555 | controller | - ' fi' 2026-03-09 19:10:03.858559 | controller | - ' # Verify RPC transport_url in DEFAULT section' 2026-03-09 19:10:03.858564 | controller | - ' rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api 2026-03-09 19:10:03.858568 | controller | -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A 2026-03-09 19:10:03.858572 | controller | 50 | grep ''transport_url'' | head -1 || true)' 2026-03-09 19:10:03.858576 | controller | - ' if [ -z "$rpc_transport_url" ]; then' 2026-03-09 19:10:03.858581 | controller | - ' echo "transport_url not found in DEFAULT section"' 2026-03-09 19:10:03.858585 | controller | - ' exit 1' 2026-03-09 19:10:03.858589 | controller | - ' fi' 2026-03-09 19:10:03.858594 | controller | - ' echo "Found RPC transport_url: $rpc_transport_url"' 2026-03-09 19:10:03.858598 | controller | - ' ' 2026-03-09 19:10:03.858602 | controller | - ' # Verify the RPC transport_url contains the correct vhost (watcher-rpc)' 2026-03-09 19:10:03.858607 | controller | - ' if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then' 2026-03-09 19:10:03.858611 | controller | - ' echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"' 2026-03-09 19:10:03.858615 | controller | - ' exit 1' 2026-03-09 19:10:03.858619 | controller | - ' fi' 2026-03-09 19:10:03.858624 | controller | - ' echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"' 2026-03-09 19:10:03.858628 | controller | - ' ' 2026-03-09 19:10:03.858634 | controller | - ' # Verify the RPC transport_url contains the correct username (watcher-rpc)' 2026-03-09 19:10:03.858638 | controller | - ' if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then' 2026-03-09 19:10:03.858643 | controller | - ' echo "RPC transport_url does not contain expected username ''watcher-rpc:''"' 2026-03-09 19:10:03.858647 | controller | - ' exit 1' 2026-03-09 19:10:03.858651 | controller | - ' fi' 2026-03-09 19:10:03.858655 | controller | - ' echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"' 2026-03-09 19:10:03.858660 | controller | - ' ' 2026-03-09 19:10:03.858664 | controller | - ' # Verify oslo_messaging_notifications section has transport_url configured' 2026-03-09 19:10:03.858668 | controller | - ' notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c 2026-03-09 19:10:03.858672 | controller | watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]'' 2026-03-09 19:10:03.858677 | controller | | grep ''transport_url'' || true)' 2026-03-09 19:10:03.858681 | controller | - ' if [ -z "$notif_transport_url" ]; then' 2026-03-09 19:10:03.858685 | controller | - ' echo "transport_url not found in oslo_messaging_notifications section"' 2026-03-09 19:10:03.858691 | controller | - ' exit 1' 2026-03-09 19:10:03.858695 | controller | - ' fi' 2026-03-09 19:10:03.858699 | controller | - ' echo "Found notifications transport_url: $notif_transport_url"' 2026-03-09 19:10:03.858704 | controller | - ' ' 2026-03-09 19:10:03.858708 | controller | - ' # Verify the notifications transport_url contains the correct vhost (watcher-notifications)' 2026-03-09 19:10:03.858714 | controller | - ' if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications''; 2026-03-09 19:10:03.858718 | controller | then' 2026-03-09 19:10:03.858722 | controller | - ' echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"' 2026-03-09 19:10:03.858727 | controller | - ' exit 1' 2026-03-09 19:10:03.858731 | controller | - ' fi' 2026-03-09 19:10:03.858735 | controller | - ' echo "Successfully verified vhost ''watcher-notifications'' in notifications 2026-03-09 19:10:03.858740 | controller | transport_url"' 2026-03-09 19:10:03.858744 | controller | - ' ' 2026-03-09 19:10:03.858748 | controller | - ' # Verify the notifications transport_url contains the correct username 2026-03-09 19:10:03.858752 | controller | (watcher-notifications)' 2026-03-09 19:10:03.858758 | controller | - ' if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:''; 2026-03-09 19:10:03.858762 | controller | then' 2026-03-09 19:10:03.858767 | controller | - ' echo "Notifications transport_url does not contain expected username 2026-03-09 19:10:03.858771 | controller | ''watcher-notifications:''"' 2026-03-09 19:10:03.858775 | controller | - ' exit 1' 2026-03-09 19:10:03.858780 | controller | - ' fi' 2026-03-09 19:10:03.858784 | controller | - ' echo "Successfully verified username ''watcher-notifications'' in notifications 2026-03-09 19:10:03.858788 | controller | transport_url"' 2026-03-09 19:10:03.858792 | controller | - ' ' 2026-03-09 19:10:03.858797 | controller | - ' exit 0' 2026-03-09 19:10:03.858801 | controller | - ' ]' 2026-03-09 19:10:03.858805 | controller | - ' logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready 2026-03-09 19:10:03.858809 | controller | watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s' 2026-03-09 19:10:03.858814 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.858819 | controller | condition met' 2026-03-09 19:10:03.858834 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl 2026-03-09 19:10:03.858840 | controller | -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''' 2026-03-09 19:10:03.858847 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + grep -q True' 2026-03-09 19:10:03.858852 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady 2026-03-09 19:10:03.858857 | controller | condition is True''' 2026-03-09 19:10:03.858861 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady 2026-03-09 19:10:03.858865 | controller | condition is True' 2026-03-09 19:10:03.858869 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.858874 | controller | -n watcher-kuttl-default -o name' 2026-03-09 19:10:03.858878 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ wc -l' 2026-03-09 19:10:03.858884 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport' 2026-03-09 19:10:03.858889 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_count=1' 2026-03-09 19:10:03.858893 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.858897 | controller | -n watcher-kuttl-default -o name' 2026-03-09 19:10:03.858902 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ wc -l' 2026-03-09 19:10:03.858906 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification' 2026-03-09 19:10:03.858910 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notification_transport_count=1' 2026-03-09 19:10:03.858914 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' 2026-03-09 19:10:03.858918 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' 2026-03-09 19:10:03.858923 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo ''Correctly found 2026-03-09 19:10:03.858929 | controller | 2 TransportURLs (separate clusters: transport and notification)''' 2026-03-09 19:10:03.858933 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs 2026-03-09 19:10:03.858937 | controller | (separate clusters: transport and notification)' 2026-03-09 19:10:03.858942 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.858946 | controller | watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}''' 2026-03-09 19:10:03.858950 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc' 2026-03-09 19:10:03.858954 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.858959 | controller | watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}''' 2026-03-09 19:10:03.858963 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc' 2026-03-09 19:10:03.858967 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' 2026-03-09 19:10:03.858972 | controller | watcher-rpc '']''' 2026-03-09 19:10:03.858976 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' 2026-03-09 19:10:03.858980 | controller | watcher-rpc '']''' 2026-03-09 19:10:03.858984 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo ''Watcher transport 2026-03-09 19:10:03.858990 | controller | has correct user (watcher-rpc) and vhost (watcher-rpc)''' 2026-03-09 19:10:03.858995 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Watcher transport has correct 2026-03-09 19:10:03.858999 | controller | user (watcher-rpc) and vhost (watcher-rpc)' 2026-03-09 19:10:03.859003 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.859009 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default 2026-03-09 19:10:03.859015 | controller | -o ''jsonpath={.spec.username}''' 2026-03-09 19:10:03.859021 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications' 2026-03-09 19:10:03.859027 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.859033 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default 2026-03-09 19:10:03.859039 | controller | -o ''jsonpath={.spec.vhost}''' 2026-03-09 19:10:03.859045 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications' 2026-03-09 19:10:03.859051 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' watcher-notifications 2026-03-09 19:10:03.859057 | controller | ''!='' watcher-notifications '']''' 2026-03-09 19:10:03.859062 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' watcher-notifications 2026-03-09 19:10:03.859067 | controller | ''!='' watcher-notifications '']''' 2026-03-09 19:10:03.859073 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo ''Notification transport 2026-03-09 19:10:03.859079 | controller | has correct user (watcher-notifications) and vhost (watcher-notifications)''' 2026-03-09 19:10:03.859083 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Notification transport 2026-03-09 19:10:03.859096 | controller | has correct user (watcher-notifications) and vhost (watcher-notifications)' 2026-03-09 19:10:03.859100 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get pods -n 2026-03-09 19:10:03.859105 | controller | watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers' 2026-03-09 19:10:03.859109 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ head -1' 2026-03-09 19:10:03.859113 | controller | - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep -v ''^$''' 2026-03-09 19:10:03.859118 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0' 2026-03-09 19:10:03.859134 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0 2026-03-09 19:10:03.859139 | controller | '']''' 2026-03-09 19:10:03.859143 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]'' 2026-03-09 19:10:03.859162 | controller | -A 50' 2026-03-09 19:10:03.859166 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep transport_url' 2026-03-09 19:10:03.859171 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-09 19:10:03.859175 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.859179 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ head -1' 2026-03-09 19:10:03.859183 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url 2026-03-09 19:10:03.859188 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-09 19:10:03.859192 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url 2026-03-09 19:10:03.859196 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'' 2026-03-09 19:10:03.859200 | controller | '']''' 2026-03-09 19:10:03.859205 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url: 2026-03-09 19:10:03.859209 | controller | transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-09 19:10:03.859213 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Found RPC transport_url: 2026-03-09 19:10:03.859217 | controller | transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1' 2026-03-09 19:10:03.859222 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-09 19:10:03.859226 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-09 19:10:03.859230 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc' 2026-03-09 19:10:03.859234 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-09 19:10:03.859239 | controller | vhost ''\''''watcher-rpc''\'''' in RPC transport_url''' 2026-03-09 19:10:03.859243 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified vhost 2026-03-09 19:10:03.859247 | controller | ''watcher-rpc'' in RPC transport_url' 2026-03-09 19:10:03.859251 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:' 2026-03-09 19:10:03.859256 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-09 19:10:03.859261 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-09 19:10:03.859267 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-09 19:10:03.859272 | controller | username ''\''''watcher-rpc''\'''' in RPC transport_url''' 2026-03-09 19:10:03.859276 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified username 2026-03-09 19:10:03.859280 | controller | ''watcher-rpc'' in RPC transport_url' 2026-03-09 19:10:03.859285 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-09 19:10:03.859293 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.859297 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]''' 2026-03-09 19:10:03.859301 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep transport_url' 2026-03-09 19:10:03.859307 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url 2026-03-09 19:10:03.859312 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-09 19:10:03.859316 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url 2026-03-09 19:10:03.859320 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'' 2026-03-09 19:10:03.859325 | controller | '']''' 2026-03-09 19:10:03.859337 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Found notifications 2026-03-09 19:10:03.859342 | controller | transport_url: transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-09 19:10:03.859346 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Found notifications transport_url: 2026-03-09 19:10:03.859351 | controller | transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1' 2026-03-09 19:10:03.859355 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-09 19:10:03.859359 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-09 19:10:03.859363 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications' 2026-03-09 19:10:03.859368 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-09 19:10:03.859372 | controller | vhost ''\''''watcher-notifications''\'''' in notifications transport_url''' 2026-03-09 19:10:03.859376 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified vhost 2026-03-09 19:10:03.859380 | controller | ''watcher-notifications'' in notifications transport_url' 2026-03-09 19:10:03.859386 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-09 19:10:03.859391 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-09 19:10:03.859396 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:' 2026-03-09 19:10:03.859400 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-09 19:10:03.859404 | controller | username ''\''''watcher-notifications''\'''' in notifications transport_url''' 2026-03-09 19:10:03.859409 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified username 2026-03-09 19:10:03.859413 | controller | ''watcher-notifications'' in notifications transport_url' 2026-03-09 19:10:03.859417 | controller | - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + exit 0' 2026-03-09 19:10:03.859421 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | running command: [sh -c 2026-03-09 19:10:03.859426 | controller | set -euxo pipefail' 2026-03-09 19:10:03.859430 | controller | - ' ' 2026-03-09 19:10:03.859434 | controller | - ' # Wait for Watcher to be Ready' 2026-03-09 19:10:03.859438 | controller | - ' kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE 2026-03-09 19:10:03.859443 | controller | --timeout=300s' 2026-03-09 19:10:03.859447 | controller | - ' ' 2026-03-09 19:10:03.859451 | controller | - ' # Verify WatcherNotificationTransportURLReady condition exists and is True' 2026-03-09 19:10:03.859455 | controller | - ' kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'' 2026-03-09 19:10:03.859460 | controller | | grep -q "True"' 2026-03-09 19:10:03.859466 | controller | - ' echo "WatcherNotificationTransportURLReady condition is True"' 2026-03-09 19:10:03.859470 | controller | - ' ' 2026-03-09 19:10:03.859474 | controller | - ' # Count TransportURL CRs - should be exactly 2 (one for messaging, one 2026-03-09 19:10:03.859479 | controller | for notifications)' 2026-03-09 19:10:03.859483 | controller | - ' transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep 2026-03-09 19:10:03.859487 | controller | "watcher-kuttl-watcher-transport" | wc -l)' 2026-03-09 19:10:03.859491 | controller | - ' notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o 2026-03-09 19:10:03.859496 | controller | name | grep "watcher-kuttl-watcher-notification" | wc -l)' 2026-03-09 19:10:03.859500 | controller | - ' ' 2026-03-09 19:10:03.859504 | controller | - ' if [ "$transport_count" -ne "1" ]; then' 2026-03-09 19:10:03.859509 | controller | - ' echo "Expected 1 watcher-transport TransportURL, found $transport_count"' 2026-03-09 19:10:03.859513 | controller | - ' exit 1' 2026-03-09 19:10:03.859518 | controller | - ' fi' 2026-03-09 19:10:03.859523 | controller | - ' ' 2026-03-09 19:10:03.859528 | controller | - ' if [ "$notification_transport_count" -ne "1" ]; then' 2026-03-09 19:10:03.859532 | controller | - ' echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"' 2026-03-09 19:10:03.859539 | controller | - ' exit 1' 2026-03-09 19:10:03.859543 | controller | - ' fi' 2026-03-09 19:10:03.859547 | controller | - ' ' 2026-03-09 19:10:03.859552 | controller | - ' echo "Correctly found 2 TransportURLs (separate clusters: transport and 2026-03-09 19:10:03.859556 | controller | notification)"' 2026-03-09 19:10:03.859560 | controller | - ' ' 2026-03-09 19:10:03.859564 | controller | - ' # Verify watcher-transport has correct user and vhost' 2026-03-09 19:10:03.859569 | controller | - ' transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-09 19:10:03.859592 | controller | -n $NAMESPACE -o jsonpath=''{.spec.username}'')' 2026-03-09 19:10:03.859597 | controller | - ' transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-09 19:10:03.859601 | controller | -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' 2026-03-09 19:10:03.859606 | controller | - ' if [ "$transport_user" != "watcher-rpc" ]; then' 2026-03-09 19:10:03.859617 | controller | - ' echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"' 2026-03-09 19:10:03.859622 | controller | - ' exit 1' 2026-03-09 19:10:03.859626 | controller | - ' fi' 2026-03-09 19:10:03.859630 | controller | - ' if [ "$transport_vhost" != "watcher-rpc" ]; then' 2026-03-09 19:10:03.859635 | controller | - ' echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"' 2026-03-09 19:10:03.859639 | controller | - ' exit 1' 2026-03-09 19:10:03.859643 | controller | - ' fi' 2026-03-09 19:10:03.859647 | controller | - ' echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"' 2026-03-09 19:10:03.859652 | controller | - ' ' 2026-03-09 19:10:03.859656 | controller | - ' # Verify notification-transport has correct user and vhost' 2026-03-09 19:10:03.859660 | controller | - ' notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-09 19:10:03.859665 | controller | -n $NAMESPACE -o jsonpath=''{.spec.username}'')' 2026-03-09 19:10:03.859669 | controller | - ' notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-09 19:10:03.859673 | controller | -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' 2026-03-09 19:10:03.859677 | controller | - ' if [ "$notif_user" != "watcher-notifications" ]; then' 2026-03-09 19:10:03.859682 | controller | - ' echo "Expected notification-transport username ''watcher-notifications'', 2026-03-09 19:10:03.859686 | controller | found ''$notif_user''"' 2026-03-09 19:10:03.859690 | controller | - ' exit 1' 2026-03-09 19:10:03.859694 | controller | - ' fi' 2026-03-09 19:10:03.859699 | controller | - ' if [ "$notif_vhost" != "watcher-notifications" ]; then' 2026-03-09 19:10:03.859705 | controller | - ' echo "Expected notification-transport vhost ''watcher-notifications'', 2026-03-09 19:10:03.859709 | controller | found ''$notif_vhost''"' 2026-03-09 19:10:03.859714 | controller | - ' exit 1' 2026-03-09 19:10:03.859718 | controller | - ' fi' 2026-03-09 19:10:03.859722 | controller | - ' echo "Notification transport has correct user (watcher-notifications) and 2026-03-09 19:10:03.859726 | controller | vhost (watcher-notifications)"' 2026-03-09 19:10:03.859731 | controller | - ' ' 2026-03-09 19:10:03.859735 | controller | - ' # Verify that watcher.conf contains the notifications transport_url' 2026-03-09 19:10:03.859739 | controller | - ' WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api" 2026-03-09 19:10:03.859743 | controller | -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)' 2026-03-09 19:10:03.859748 | controller | - ' if [ -z "${WATCHER_API_POD}" ]; then' 2026-03-09 19:10:03.859752 | controller | - ' echo "No watcher-api pod found"' 2026-03-09 19:10:03.859756 | controller | - ' exit 1' 2026-03-09 19:10:03.859760 | controller | - ' fi' 2026-03-09 19:10:03.859765 | controller | - ' # Verify RPC transport_url in DEFAULT section' 2026-03-09 19:10:03.859769 | controller | - ' rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api 2026-03-09 19:10:03.859773 | controller | -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A 2026-03-09 19:10:03.859778 | controller | 50 | grep ''transport_url'' | head -1 || true)' 2026-03-09 19:10:03.859782 | controller | - ' if [ -z "$rpc_transport_url" ]; then' 2026-03-09 19:10:03.859786 | controller | - ' echo "transport_url not found in DEFAULT section"' 2026-03-09 19:10:03.859790 | controller | - ' exit 1' 2026-03-09 19:10:03.859795 | controller | - ' fi' 2026-03-09 19:10:03.859799 | controller | - ' echo "Found RPC transport_url: $rpc_transport_url"' 2026-03-09 19:10:03.859803 | controller | - ' ' 2026-03-09 19:10:03.859807 | controller | - ' # Verify the RPC transport_url contains the correct vhost (watcher-rpc)' 2026-03-09 19:10:03.859812 | controller | - ' if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then' 2026-03-09 19:10:03.859816 | controller | - ' echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"' 2026-03-09 19:10:03.859820 | controller | - ' exit 1' 2026-03-09 19:10:03.859824 | controller | - ' fi' 2026-03-09 19:10:03.859829 | controller | - ' echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"' 2026-03-09 19:10:03.859833 | controller | - ' ' 2026-03-09 19:10:03.859837 | controller | - ' # Verify the RPC transport_url contains the correct username (watcher-rpc)' 2026-03-09 19:10:03.859841 | controller | - ' if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then' 2026-03-09 19:10:03.859846 | controller | - ' echo "RPC transport_url does not contain expected username ''watcher-rpc:''"' 2026-03-09 19:10:03.859850 | controller | - ' exit 1' 2026-03-09 19:10:03.859854 | controller | - ' fi' 2026-03-09 19:10:03.859858 | controller | - ' echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"' 2026-03-09 19:10:03.859863 | controller | - ' ' 2026-03-09 19:10:03.859867 | controller | - ' # Verify oslo_messaging_notifications section has transport_url configured' 2026-03-09 19:10:03.859871 | controller | - ' notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c 2026-03-09 19:10:03.859875 | controller | watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]'' 2026-03-09 19:10:03.859880 | controller | | grep ''transport_url'' || true)' 2026-03-09 19:10:03.859884 | controller | - ' if [ -z "$notif_transport_url" ]; then' 2026-03-09 19:10:03.859888 | controller | - ' echo "transport_url not found in oslo_messaging_notifications section"' 2026-03-09 19:10:03.859892 | controller | - ' exit 1' 2026-03-09 19:10:03.859897 | controller | - ' fi' 2026-03-09 19:10:03.859903 | controller | - ' echo "Found notifications transport_url: $notif_transport_url"' 2026-03-09 19:10:03.859907 | controller | - ' ' 2026-03-09 19:10:03.859913 | controller | - ' # Verify the notifications transport_url contains the correct vhost (watcher-notifications)' 2026-03-09 19:10:03.859919 | controller | - ' if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications''; 2026-03-09 19:10:03.859923 | controller | then' 2026-03-09 19:10:03.859934 | controller | - ' echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"' 2026-03-09 19:10:03.859939 | controller | - ' exit 1' 2026-03-09 19:10:03.859943 | controller | - ' fi' 2026-03-09 19:10:03.859948 | controller | - ' echo "Successfully verified vhost ''watcher-notifications'' in notifications 2026-03-09 19:10:03.859952 | controller | transport_url"' 2026-03-09 19:10:03.859956 | controller | - ' ' 2026-03-09 19:10:03.859961 | controller | - ' # Verify the notifications transport_url contains the correct username 2026-03-09 19:10:03.859965 | controller | (watcher-notifications)' 2026-03-09 19:10:03.859969 | controller | - ' if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:''; 2026-03-09 19:10:03.859974 | controller | then' 2026-03-09 19:10:03.859978 | controller | - ' echo "Notifications transport_url does not contain expected username 2026-03-09 19:10:03.859982 | controller | ''watcher-notifications:''"' 2026-03-09 19:10:03.859986 | controller | - ' exit 1' 2026-03-09 19:10:03.859991 | controller | - ' fi' 2026-03-09 19:10:03.859995 | controller | - ' echo "Successfully verified username ''watcher-notifications'' in notifications 2026-03-09 19:10:03.859999 | controller | transport_url"' 2026-03-09 19:10:03.860003 | controller | - ' ' 2026-03-09 19:10:03.860009 | controller | - ' exit 0' 2026-03-09 19:10:03.860015 | controller | - ' ]' 2026-03-09 19:10:03.860021 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready 2026-03-09 19:10:03.860027 | controller | watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s' 2026-03-09 19:10:03.860034 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.860039 | controller | condition met' 2026-03-09 19:10:03.860045 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl 2026-03-09 19:10:03.860051 | controller | -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''' 2026-03-09 19:10:03.860057 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + grep -q True' 2026-03-09 19:10:03.860062 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady 2026-03-09 19:10:03.860068 | controller | condition is True''' 2026-03-09 19:10:03.860073 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady 2026-03-09 19:10:03.860079 | controller | condition is True' 2026-03-09 19:10:03.860083 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport' 2026-03-09 19:10:03.860090 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.860095 | controller | -n watcher-kuttl-default -o name' 2026-03-09 19:10:03.860099 | controller | - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ wc -l' 2026-03-09 19:10:03.860103 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_count=1' 2026-03-09 19:10:03.860107 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ wc -l' 2026-03-09 19:10:03.860112 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification' 2026-03-09 19:10:03.860116 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.860120 | controller | -n watcher-kuttl-default -o name' 2026-03-09 19:10:03.860124 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notification_transport_count=1' 2026-03-09 19:10:03.860129 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' 2026-03-09 19:10:03.860133 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' 2026-03-09 19:10:03.860139 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo ''Correctly found 2026-03-09 19:10:03.860143 | controller | 2 TransportURLs (separate clusters: transport and notification)''' 2026-03-09 19:10:03.860161 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs 2026-03-09 19:10:03.860166 | controller | (separate clusters: transport and notification)' 2026-03-09 19:10:03.860170 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.860174 | controller | watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}''' 2026-03-09 19:10:03.860179 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc' 2026-03-09 19:10:03.860183 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.860187 | controller | watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}''' 2026-03-09 19:10:03.860191 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc' 2026-03-09 19:10:03.860195 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' 2026-03-09 19:10:03.860200 | controller | watcher-rpc '']''' 2026-03-09 19:10:03.860204 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' 2026-03-09 19:10:03.860208 | controller | watcher-rpc '']''' 2026-03-09 19:10:03.860212 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo ''Watcher transport 2026-03-09 19:10:03.860217 | controller | has correct user (watcher-rpc) and vhost (watcher-rpc)''' 2026-03-09 19:10:03.860221 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Watcher transport has correct 2026-03-09 19:10:03.860225 | controller | user (watcher-rpc) and vhost (watcher-rpc)' 2026-03-09 19:10:03.860230 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.860243 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default 2026-03-09 19:10:03.860247 | controller | -o ''jsonpath={.spec.username}''' 2026-03-09 19:10:03.860252 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications' 2026-03-09 19:10:03.860258 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-09 19:10:03.860262 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default 2026-03-09 19:10:03.860266 | controller | -o ''jsonpath={.spec.vhost}''' 2026-03-09 19:10:03.860271 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications' 2026-03-09 19:10:03.860275 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' watcher-notifications 2026-03-09 19:10:03.860279 | controller | ''!='' watcher-notifications '']''' 2026-03-09 19:10:03.860283 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' watcher-notifications 2026-03-09 19:10:03.860288 | controller | ''!='' watcher-notifications '']''' 2026-03-09 19:10:03.860292 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo ''Notification transport 2026-03-09 19:10:03.860296 | controller | has correct user (watcher-notifications) and vhost (watcher-notifications)''' 2026-03-09 19:10:03.860301 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Notification transport 2026-03-09 19:10:03.860305 | controller | has correct user (watcher-notifications) and vhost (watcher-notifications)' 2026-03-09 19:10:03.860309 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep -v ''^$''' 2026-03-09 19:10:03.860314 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ head -1' 2026-03-09 19:10:03.860318 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get pods -n 2026-03-09 19:10:03.860322 | controller | watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers' 2026-03-09 19:10:03.860328 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0' 2026-03-09 19:10:03.860333 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0 2026-03-09 19:10:03.860337 | controller | '']''' 2026-03-09 19:10:03.860341 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]'' 2026-03-09 19:10:03.860346 | controller | -A 50' 2026-03-09 19:10:03.860350 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep transport_url' 2026-03-09 19:10:03.860354 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-09 19:10:03.860358 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.860363 | controller | - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ head -1' 2026-03-09 19:10:03.860367 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url 2026-03-09 19:10:03.860371 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-09 19:10:03.860375 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url 2026-03-09 19:10:03.860380 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'' 2026-03-09 19:10:03.860384 | controller | '']''' 2026-03-09 19:10:03.860388 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url: 2026-03-09 19:10:03.860393 | controller | transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-09 19:10:03.860399 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Found RPC transport_url: 2026-03-09 19:10:03.860403 | controller | transport_url = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1' 2026-03-09 19:10:03.860407 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-09 19:10:03.860413 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-09 19:10:03.860417 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc' 2026-03-09 19:10:03.860422 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-09 19:10:03.860427 | controller | vhost ''\''''watcher-rpc''\'''' in RPC transport_url''' 2026-03-09 19:10:03.860432 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified vhost 2026-03-09 19:10:03.860436 | controller | ''watcher-rpc'' in RPC transport_url' 2026-03-09 19:10:03.860440 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-09 19:10:03.860445 | controller | = rabbit://watcher-rpc:cwlQnXc0itUpXEuIh2kc9h8KrdOF3Syc@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-09 19:10:03.860449 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:' 2026-03-09 19:10:03.860453 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-09 19:10:03.860465 | controller | username ''\''''watcher-rpc''\'''' in RPC transport_url''' 2026-03-09 19:10:03.860470 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified username 2026-03-09 19:10:03.860474 | controller | ''watcher-rpc'' in RPC transport_url' 2026-03-09 19:10:03.860479 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-09 19:10:03.860483 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.860488 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ grep transport_url' 2026-03-09 19:10:03.860495 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]''' 2026-03-09 19:10:03.860499 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url 2026-03-09 19:10:03.860503 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-09 19:10:03.860508 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url 2026-03-09 19:10:03.860512 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'' 2026-03-09 19:10:03.860516 | controller | '']''' 2026-03-09 19:10:03.860521 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Found notifications 2026-03-09 19:10:03.860525 | controller | transport_url: transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-09 19:10:03.860529 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Found notifications transport_url: 2026-03-09 19:10:03.860533 | controller | transport_url = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1' 2026-03-09 19:10:03.860538 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications' 2026-03-09 19:10:03.860542 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-09 19:10:03.860546 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-09 19:10:03.860550 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-09 19:10:03.860555 | controller | vhost ''\''''watcher-notifications''\'''' in notifications transport_url''' 2026-03-09 19:10:03.860559 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified vhost 2026-03-09 19:10:03.860563 | controller | ''watcher-notifications'' in notifications transport_url' 2026-03-09 19:10:03.860567 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:' 2026-03-09 19:10:03.860572 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-09 19:10:03.860576 | controller | = rabbit://watcher-notifications:yM_YorUcGVepZZD3yTsTRS1M84Yn_wfB@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-09 19:10:03.860580 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-09 19:10:03.860585 | controller | username ''\''''watcher-notifications''\'''' in notifications transport_url''' 2026-03-09 19:10:03.860589 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified username 2026-03-09 19:10:03.860593 | controller | ''watcher-notifications'' in notifications transport_url' 2026-03-09 19:10:03.860597 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + exit 0' 2026-03-09 19:10:03.860602 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | test step completed 1-deploy' 2026-03-09 19:10:03.860606 | controller | - ' logger.go:42: 19:05:16 | watcher-rmquser/2-cleanup-watcher | starting test 2026-03-09 19:10:03.860610 | controller | step 2-cleanup-watcher' 2026-03-09 19:10:03.860614 | controller | - ' logger.go:42: 19:05:22 | watcher-rmquser/2-cleanup-watcher | test step completed 2026-03-09 19:10:03.860619 | controller | 2-cleanup-watcher' 2026-03-09 19:10:03.860623 | controller | - ' logger.go:42: 19:05:22 | watcher-rmquser | skipping kubernetes event logging' 2026-03-09 19:10:03.860627 | controller | - === CONT kuttl/harness/watcher 2026-03-09 19:10:03.860632 | controller | - ' logger.go:42: 19:05:22 | watcher | Skipping creation of user-supplied namespace: 2026-03-09 19:10:03.860637 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.860642 | controller | - ' logger.go:42: 19:05:22 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher' 2026-03-09 19:10:03.860646 | controller | - ' logger.go:42: 19:05:22 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher' 2026-03-09 19:10:03.860650 | controller | - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | starting test step 2026-03-09 19:10:03.860654 | controller | 1-deploy-with-defaults' 2026-03-09 19:10:03.860659 | controller | - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.860663 | controller | created' 2026-03-09 19:10:03.860674 | controller | - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | running command: 2026-03-09 19:10:03.860678 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.860683 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.860687 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.860691 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.860696 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.860700 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.860704 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.860708 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.860713 | controller | ]' 2026-03-09 19:10:03.860717 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.860721 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.860727 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.860732 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-09 19:10:03.860736 | controller | == 2 ]' 2026-03-09 19:10:03.860740 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.860745 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-09 19:10:03.860749 | controller | -ge 1 ]' 2026-03-09 19:10:03.860753 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.860757 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.860762 | controller | == "" ]; then' 2026-03-09 19:10:03.860766 | controller | - ' exit 0' 2026-03-09 19:10:03.860770 | controller | - ' fi' 2026-03-09 19:10:03.860774 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.860779 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.860783 | controller | - ' counter=0' 2026-03-09 19:10:03.860787 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.860791 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.860796 | controller | - ' echo ${i}' 2026-03-09 19:10:03.860800 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.860806 | controller | - ' fi' 2026-03-09 19:10:03.860811 | controller | - ' done' 2026-03-09 19:10:03.860815 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.860819 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.860824 | controller | - ' exit 1' 2026-03-09 19:10:03.860828 | controller | - ' else' 2026-03-09 19:10:03.860832 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.860836 | controller | - ' fi' 2026-03-09 19:10:03.860841 | controller | - ' ]' 2026-03-09 19:10:03.860847 | controller | - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.860851 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.860855 | controller | - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.860860 | controller | - ' logger.go:42: 19:05:24 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.860864 | controller | - ' logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | running command: 2026-03-09 19:10:03.860868 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.860872 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.860877 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.860881 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.860885 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.860889 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.860894 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.860898 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.860902 | controller | ]' 2026-03-09 19:10:03.860906 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.860911 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.860915 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.860919 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-09 19:10:03.860923 | controller | == 2 ]' 2026-03-09 19:10:03.860928 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.860932 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-09 19:10:03.860936 | controller | -ge 1 ]' 2026-03-09 19:10:03.860940 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.860945 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.860956 | controller | == "" ]; then' 2026-03-09 19:10:03.860961 | controller | - ' exit 0' 2026-03-09 19:10:03.860965 | controller | - ' fi' 2026-03-09 19:10:03.860969 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.860974 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.860978 | controller | - ' counter=0' 2026-03-09 19:10:03.860982 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.860987 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.860991 | controller | - ' echo ${i}' 2026-03-09 19:10:03.860995 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.860999 | controller | - ' fi' 2026-03-09 19:10:03.861004 | controller | - ' done' 2026-03-09 19:10:03.861009 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.861015 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.861021 | controller | - ' exit 1' 2026-03-09 19:10:03.861027 | controller | - ' else' 2026-03-09 19:10:03.861033 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.861039 | controller | - ' fi' 2026-03-09 19:10:03.861044 | controller | - ' ]' 2026-03-09 19:10:03.861048 | controller | - ' logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.861052 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.861059 | controller | - ' logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.861063 | controller | - ' logger.go:42: 19:05:28 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']''' 2026-03-09 19:10:03.861067 | controller | - ' logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | running command: 2026-03-09 19:10:03.861072 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.861076 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.861080 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.861084 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.861089 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.861093 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.861097 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.861102 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.861106 | controller | ]' 2026-03-09 19:10:03.861110 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861114 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.861119 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861123 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-09 19:10:03.861127 | controller | == 2 ]' 2026-03-09 19:10:03.861131 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861136 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-09 19:10:03.861140 | controller | -ge 1 ]' 2026-03-09 19:10:03.861156 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.861163 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.861167 | controller | == "" ]; then' 2026-03-09 19:10:03.861171 | controller | - ' exit 0' 2026-03-09 19:10:03.861176 | controller | - ' fi' 2026-03-09 19:10:03.861180 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.861184 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.861188 | controller | - ' counter=0' 2026-03-09 19:10:03.861193 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.861197 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.861201 | controller | - ' echo ${i}' 2026-03-09 19:10:03.861206 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.861210 | controller | - ' fi' 2026-03-09 19:10:03.861214 | controller | - ' done' 2026-03-09 19:10:03.861218 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.861223 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.861227 | controller | - ' exit 1' 2026-03-09 19:10:03.861231 | controller | - ' else' 2026-03-09 19:10:03.861235 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.861240 | controller | - ' fi' 2026-03-09 19:10:03.861244 | controller | - ' ]' 2026-03-09 19:10:03.861248 | controller | - ' logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.861252 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.861257 | controller | - ' logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.861261 | controller | - ' logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.861267 | controller | - ' logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-09 19:10:03.861271 | controller | - ' logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-09 19:10:03.861276 | controller | - ' logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.861288 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.861294 | controller | - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' 2026-03-09 19:10:03.861299 | controller | - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861303 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.861307 | controller | - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.861312 | controller | == 5be153439946473bb5b269e3a0a3e136 '']''' 2026-03-09 19:10:03.861316 | controller | - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861320 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.861326 | controller | - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + ''['' -n '''' '']''' 2026-03-09 19:10:03.861330 | controller | - ' logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | running command: 2026-03-09 19:10:03.861335 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.861339 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.861343 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.861348 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.861352 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.861356 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.861361 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.861365 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.861369 | controller | ]' 2026-03-09 19:10:03.861373 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861378 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.861382 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861386 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-09 19:10:03.861391 | controller | == 2 ]' 2026-03-09 19:10:03.861395 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861399 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-09 19:10:03.861403 | controller | -ge 1 ]' 2026-03-09 19:10:03.861407 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.861412 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.861416 | controller | == "" ]; then' 2026-03-09 19:10:03.861420 | controller | - ' exit 0' 2026-03-09 19:10:03.861425 | controller | - ' fi' 2026-03-09 19:10:03.861429 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.861433 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.861437 | controller | - ' counter=0' 2026-03-09 19:10:03.861442 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.861446 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.861452 | controller | - ' echo ${i}' 2026-03-09 19:10:03.861457 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.861461 | controller | - ' fi' 2026-03-09 19:10:03.861465 | controller | - ' done' 2026-03-09 19:10:03.861469 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.861474 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.861478 | controller | - ' exit 1' 2026-03-09 19:10:03.861482 | controller | - ' else' 2026-03-09 19:10:03.861488 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.861492 | controller | - ' fi' 2026-03-09 19:10:03.861497 | controller | - ' ]' 2026-03-09 19:10:03.861501 | controller | - ' logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.861505 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.861509 | controller | - ' logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.861514 | controller | - ' logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.861518 | controller | - ' logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-09 19:10:03.861522 | controller | - ' logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-09 19:10:03.861526 | controller | - ' logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.861531 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.861535 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' 2026-03-09 19:10:03.861539 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861543 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.861555 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.861559 | controller | == 5be153439946473bb5b269e3a0a3e136 '']''' 2026-03-09 19:10:03.861564 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861568 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.861572 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.861577 | controller | '']''' 2026-03-09 19:10:03.861581 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.861585 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861590 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-09 19:10:03.861594 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' 2026-03-09 19:10:03.861598 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.861602 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.861607 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861611 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.861615 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile 2026-03-09 19:10:03.861619 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-09 19:10:03.861623 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.861629 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.861634 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861638 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.861642 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name 2026-03-09 19:10:03.861648 | controller | = regionOne''' 2026-03-09 19:10:03.861654 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' 2026-03-09 19:10:03.861660 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get pods -n 2026-03-09 19:10:03.861665 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.861670 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' 2026-03-09 19:10:03.861676 | controller | '']''' 2026-03-09 19:10:03.861681 | controller | - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + exit 0' 2026-03-09 19:10:03.861686 | controller | - ' logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | running command: 2026-03-09 19:10:03.861692 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.861698 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.861704 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.861709 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.861718 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.861724 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.861729 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.861735 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.861741 | controller | ]' 2026-03-09 19:10:03.861747 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861753 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.861759 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861764 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-09 19:10:03.861770 | controller | == 2 ]' 2026-03-09 19:10:03.861775 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.861781 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-09 19:10:03.861787 | controller | -ge 1 ]' 2026-03-09 19:10:03.861792 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.861798 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.861804 | controller | == "" ]; then' 2026-03-09 19:10:03.861810 | controller | - ' exit 0' 2026-03-09 19:10:03.861816 | controller | - ' fi' 2026-03-09 19:10:03.861822 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.861826 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.861830 | controller | - ' counter=0' 2026-03-09 19:10:03.861834 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.861848 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.861853 | controller | - ' echo ${i}' 2026-03-09 19:10:03.861857 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.861862 | controller | - ' fi' 2026-03-09 19:10:03.861866 | controller | - ' done' 2026-03-09 19:10:03.861873 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.861877 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.861882 | controller | - ' exit 1' 2026-03-09 19:10:03.861886 | controller | - ' else' 2026-03-09 19:10:03.861893 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.861897 | controller | - ' fi' 2026-03-09 19:10:03.861902 | controller | - ' ]' 2026-03-09 19:10:03.861906 | controller | - ' logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.861910 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.861915 | controller | - ' logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.861919 | controller | - ' logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.861923 | controller | - ' logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-09 19:10:03.861927 | controller | - ' logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-09 19:10:03.861932 | controller | - ' logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.861936 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.861940 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' 2026-03-09 19:10:03.861945 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861949 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.861953 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.861957 | controller | == 5be153439946473bb5b269e3a0a3e136 '']''' 2026-03-09 19:10:03.861962 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861966 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.861970 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.861975 | controller | '']''' 2026-03-09 19:10:03.861979 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.861986 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' 2026-03-09 19:10:03.861990 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.861995 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-09 19:10:03.861999 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.862003 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.862009 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862015 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.862021 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile 2026-03-09 19:10:03.862027 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-09 19:10:03.862033 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.862039 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862044 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.862050 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.862058 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name 2026-03-09 19:10:03.862064 | controller | = regionOne''' 2026-03-09 19:10:03.862070 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' 2026-03-09 19:10:03.862075 | controller | - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get pods -n 2026-03-09 19:10:03.862079 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.862083 | controller | - ' logger.go:42: 19:05:46 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' 2026-03-09 19:10:03.862087 | controller | '']''' 2026-03-09 19:10:03.862094 | controller | - ' logger.go:42: 19:05:46 | watcher/1-deploy-with-defaults | + exit 0' 2026-03-09 19:10:03.862103 | controller | - ' logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | running command: 2026-03-09 19:10:03.862118 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.862125 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.862131 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.862162 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.862171 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.862176 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.862180 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.862184 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.862189 | controller | ]' 2026-03-09 19:10:03.862193 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.862198 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.862202 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.862206 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-09 19:10:03.862211 | controller | == 2 ]' 2026-03-09 19:10:03.862215 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.862223 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-09 19:10:03.862228 | controller | -ge 1 ]' 2026-03-09 19:10:03.862232 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.862236 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.862241 | controller | == "" ]; then' 2026-03-09 19:10:03.862245 | controller | - ' exit 0' 2026-03-09 19:10:03.862249 | controller | - ' fi' 2026-03-09 19:10:03.862253 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.862257 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.862262 | controller | - ' counter=0' 2026-03-09 19:10:03.862266 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.862270 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.862275 | controller | - ' echo ${i}' 2026-03-09 19:10:03.862279 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.862283 | controller | - ' fi' 2026-03-09 19:10:03.862287 | controller | - ' done' 2026-03-09 19:10:03.862291 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.862296 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.862300 | controller | - ' exit 1' 2026-03-09 19:10:03.862304 | controller | - ' else' 2026-03-09 19:10:03.862308 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.862315 | controller | - ' fi' 2026-03-09 19:10:03.862319 | controller | - ' ]' 2026-03-09 19:10:03.862324 | controller | - ' logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.862328 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.862332 | controller | - ' logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.862336 | controller | - ' logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.862341 | controller | - ' logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.862345 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.862349 | controller | - ' logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-09 19:10:03.862354 | controller | - ' logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-09 19:10:03.862358 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' 2026-03-09 19:10:03.862362 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862366 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.862372 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.862377 | controller | == 5be153439946473bb5b269e3a0a3e136 '']''' 2026-03-09 19:10:03.862381 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862385 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.862389 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.862394 | controller | '']''' 2026-03-09 19:10:03.862398 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' 2026-03-09 19:10:03.862402 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.862406 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862410 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-09 19:10:03.862415 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.862419 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile 2026-03-09 19:10:03.862423 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-09 19:10:03.862435 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.862439 | controller | - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862443 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.862448 | controller | - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.862452 | controller | - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.862456 | controller | - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862460 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.862465 | controller | - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name 2026-03-09 19:10:03.862469 | controller | = regionOne''' 2026-03-09 19:10:03.862473 | controller | - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' 2026-03-09 19:10:03.862479 | controller | - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ oc get pods -n 2026-03-09 19:10:03.862483 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.862488 | controller | - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' 2026-03-09 19:10:03.862492 | controller | '']''' 2026-03-09 19:10:03.862496 | controller | - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + exit 0' 2026-03-09 19:10:03.862500 | controller | - ' logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | running command: 2026-03-09 19:10:03.862504 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.862509 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.862513 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.862517 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.862522 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.862526 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.862530 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.862534 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.862539 | controller | ]' 2026-03-09 19:10:03.862543 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.862547 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-09 19:10:03.862551 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.862556 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-09 19:10:03.862560 | controller | == 2 ]' 2026-03-09 19:10:03.862564 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-09 19:10:03.862569 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-09 19:10:03.862573 | controller | -ge 1 ]' 2026-03-09 19:10:03.862577 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.862581 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.862585 | controller | == "" ]; then' 2026-03-09 19:10:03.862590 | controller | - ' exit 0' 2026-03-09 19:10:03.862594 | controller | - ' fi' 2026-03-09 19:10:03.862598 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.862602 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.862607 | controller | - ' counter=0' 2026-03-09 19:10:03.862611 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.862615 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.862619 | controller | - ' echo ${i}' 2026-03-09 19:10:03.862624 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.862628 | controller | - ' fi' 2026-03-09 19:10:03.862632 | controller | - ' done' 2026-03-09 19:10:03.862636 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.862641 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.862645 | controller | - ' exit 1' 2026-03-09 19:10:03.862649 | controller | - ' else' 2026-03-09 19:10:03.862653 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.862657 | controller | - ' fi' 2026-03-09 19:10:03.862662 | controller | - ' ]' 2026-03-09 19:10:03.862666 | controller | - ' logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.862670 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.862676 | controller | - ' logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.862680 | controller | - ' logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.862686 | controller | - ' logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-09 19:10:03.862690 | controller | - ' logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-09 19:10:03.862695 | controller | - ' logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.862706 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-09 19:10:03.862711 | controller | - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' 2026-03-09 19:10:03.862715 | controller | - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862719 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.862724 | controller | - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 2026-03-09 19:10:03.862728 | controller | == 5be153439946473bb5b269e3a0a3e136 '']''' 2026-03-09 19:10:03.862732 | controller | - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862736 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.862741 | controller | - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.862745 | controller | '']''' 2026-03-09 19:10:03.862749 | controller | - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.862755 | controller | - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' 2026-03-09 19:10:03.862761 | controller | - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862766 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-09 19:10:03.862772 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-09 19:10:03.862778 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile 2026-03-09 19:10:03.862784 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-09 19:10:03.862790 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.862795 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862820 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.862826 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.862832 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name 2026-03-09 19:10:03.862838 | controller | = regionOne''' 2026-03-09 19:10:03.862844 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-09 19:10:03.862849 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-09 19:10:03.862855 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-09 19:10:03.862861 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' 2026-03-09 19:10:03.862866 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get pods -n 2026-03-09 19:10:03.862872 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.862878 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' 2026-03-09 19:10:03.862887 | controller | '']''' 2026-03-09 19:10:03.862893 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + exit 0' 2026-03-09 19:10:03.862898 | controller | - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | test step completed 2026-03-09 19:10:03.862904 | controller | 1-deploy-with-defaults' 2026-03-09 19:10:03.862910 | controller | - ' logger.go:42: 19:05:58 | watcher/2-cleanup-watcher | starting test step 2-cleanup-watcher' 2026-03-09 19:10:03.862916 | controller | - ' logger.go:42: 19:05:58 | watcher/2-cleanup-watcher | test step completed 2-cleanup-watcher' 2026-03-09 19:10:03.862921 | controller | - ' logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | starting test 2026-03-09 19:10:03.862926 | controller | step 3-precreate-mariadbaccount' 2026-03-09 19:10:03.862933 | controller | - ' logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | MariaDBAccount:watcher-kuttl-default/watcher-precreated 2026-03-09 19:10:03.862937 | controller | created' 2026-03-09 19:10:03.862941 | controller | - ' logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | test step completed 2026-03-09 19:10:03.862946 | controller | 3-precreate-mariadbaccount' 2026-03-09 19:10:03.862950 | controller | - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | starting 2026-03-09 19:10:03.862954 | controller | test step 4-deploy-with-precreated-account' 2026-03-09 19:10:03.862958 | controller | - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | Secret:watcher-kuttl-default/custom-prometheus-config 2026-03-09 19:10:03.862963 | controller | created' 2026-03-09 19:10:03.862967 | controller | - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.862971 | controller | created' 2026-03-09 19:10:03.862976 | controller | - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.862980 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.862984 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.862989 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.863001 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.863006 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.863010 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.863015 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.863019 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.863023 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.863027 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.863032 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.863036 | controller | == 1 ]' 2026-03-09 19:10:03.863040 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.863045 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.863049 | controller | - ' else' 2026-03-09 19:10:03.863053 | controller | - ' exit 1' 2026-03-09 19:10:03.863057 | controller | - ' fi' 2026-03-09 19:10:03.863062 | controller | - ' ]' 2026-03-09 19:10:03.863066 | controller | - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.863070 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.863074 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.863079 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.863083 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.863089 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.863093 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.863098 | controller | -1' 2026-03-09 19:10:03.863102 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.863106 | controller | -v ''^$''' 2026-03-09 19:10:03.863110 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.863115 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.863119 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.863123 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.863127 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.863133 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.863138 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.863142 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | error: 2026-03-09 19:10:03.863167 | controller | Internal error occurred: error executing command in container: container is not 2026-03-09 19:10:03.863172 | controller | created or running' 2026-03-09 19:10:03.863176 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ echo' 2026-03-09 19:10:03.863181 | controller | - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.863185 | controller | 0 == 1 '']''' 2026-03-09 19:10:03.863189 | controller | - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.863193 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.863197 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.863202 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.863206 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.863210 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.863214 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.863219 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.863223 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.863227 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.863231 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.863236 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.863240 | controller | == 1 ]' 2026-03-09 19:10:03.863244 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.863248 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.863253 | controller | - ' else' 2026-03-09 19:10:03.863257 | controller | - ' exit 1' 2026-03-09 19:10:03.863261 | controller | - ' fi' 2026-03-09 19:10:03.863265 | controller | - ' ]' 2026-03-09 19:10:03.863270 | controller | - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.863274 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.863278 | controller | - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.863290 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.863295 | controller | - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.863301 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.863305 | controller | - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.863310 | controller | -1' 2026-03-09 19:10:03.863314 | controller | - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.863318 | controller | -v ''^$''' 2026-03-09 19:10:03.863322 | controller | - ' logger.go:42: 19:06:01 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.863327 | controller | - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.863331 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.863335 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.863339 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.863344 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.863348 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.863352 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.863356 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.863361 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.863365 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.863369 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.863374 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.863378 | controller | == 1 ]' 2026-03-09 19:10:03.863382 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.863386 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.863391 | controller | - ' else' 2026-03-09 19:10:03.863395 | controller | - ' exit 1' 2026-03-09 19:10:03.863399 | controller | - ' fi' 2026-03-09 19:10:03.863403 | controller | - ' ]' 2026-03-09 19:10:03.863408 | controller | - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.863412 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.863416 | controller | - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.863421 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.863425 | controller | - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.863429 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.863433 | controller | - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.863438 | controller | -1' 2026-03-09 19:10:03.863442 | controller | - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.863446 | controller | -v ''^$''' 2026-03-09 19:10:03.863450 | controller | - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.863455 | controller | - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.863459 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.863465 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.863470 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.863474 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.863478 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.863483 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.863489 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.863493 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.863497 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.863502 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.863506 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.863510 | controller | == 1 ]' 2026-03-09 19:10:03.863514 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.863519 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.863523 | controller | - ' else' 2026-03-09 19:10:03.863527 | controller | - ' exit 1' 2026-03-09 19:10:03.863535 | controller | - ' fi' 2026-03-09 19:10:03.863539 | controller | - ' ]' 2026-03-09 19:10:03.863543 | controller | - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.863548 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.863552 | controller | - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.863556 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.863568 | controller | - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.863595 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.863600 | controller | - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.863604 | controller | -1' 2026-03-09 19:10:03.863609 | controller | - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.863613 | controller | -v ''^$''' 2026-03-09 19:10:03.863617 | controller | - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.863622 | controller | - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.863628 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.863632 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.863636 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.863641 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.863646 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.863652 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.863658 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.863664 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.863670 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.863676 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.863682 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.863688 | controller | == 1 ]' 2026-03-09 19:10:03.863697 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.863703 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.863709 | controller | - ' else' 2026-03-09 19:10:03.863713 | controller | - ' exit 1' 2026-03-09 19:10:03.863717 | controller | - ' fi' 2026-03-09 19:10:03.863721 | controller | - ' ]' 2026-03-09 19:10:03.863726 | controller | - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.863732 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.863737 | controller | - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.863741 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.863745 | controller | - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.863749 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.863755 | controller | - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.863761 | controller | -1' 2026-03-09 19:10:03.863767 | controller | - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.863773 | controller | -v ''^$''' 2026-03-09 19:10:03.863778 | controller | - ' logger.go:42: 19:06:05 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.863783 | controller | - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.863789 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.863794 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.863799 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.863805 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.863810 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.863815 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.863821 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.863826 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.863832 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.863838 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.863844 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.863850 | controller | == 1 ]' 2026-03-09 19:10:03.863855 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.863861 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.863867 | controller | - ' else' 2026-03-09 19:10:03.863872 | controller | - ' exit 1' 2026-03-09 19:10:03.863876 | controller | - ' fi' 2026-03-09 19:10:03.863881 | controller | - ' ]' 2026-03-09 19:10:03.863885 | controller | - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.863889 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.863893 | controller | - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.863898 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.863902 | controller | - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.863915 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.863920 | controller | - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.863924 | controller | -1' 2026-03-09 19:10:03.863929 | controller | - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.863933 | controller | -v ''^$''' 2026-03-09 19:10:03.863937 | controller | - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.863941 | controller | - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.863946 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.863952 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.863957 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.863963 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.863968 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.863972 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.863977 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.863981 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.863985 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.863989 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.863994 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.863998 | controller | == 1 ]' 2026-03-09 19:10:03.864002 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.864007 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.864011 | controller | - ' else' 2026-03-09 19:10:03.864015 | controller | - ' exit 1' 2026-03-09 19:10:03.864019 | controller | - ' fi' 2026-03-09 19:10:03.864023 | controller | - ' ]' 2026-03-09 19:10:03.864028 | controller | - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.864032 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.864036 | controller | - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.864041 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.864045 | controller | - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.864049 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.864055 | controller | - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.864059 | controller | -1' 2026-03-09 19:10:03.864064 | controller | - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.864068 | controller | -v ''^$''' 2026-03-09 19:10:03.864072 | controller | - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.864076 | controller | - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.864081 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.864085 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.864089 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.864093 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.864098 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.864102 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.864106 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.864110 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.864115 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.864119 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.864123 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.864127 | controller | == 1 ]' 2026-03-09 19:10:03.864133 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.864137 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.864142 | controller | - ' else' 2026-03-09 19:10:03.864161 | controller | - ' exit 1' 2026-03-09 19:10:03.864166 | controller | - ' fi' 2026-03-09 19:10:03.864170 | controller | - ' ]' 2026-03-09 19:10:03.864174 | controller | - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.864179 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.864183 | controller | - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.864187 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.864191 | controller | - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.864203 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.864208 | controller | - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.864212 | controller | -1' 2026-03-09 19:10:03.864216 | controller | - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.864220 | controller | -v ''^$''' 2026-03-09 19:10:03.864225 | controller | - ' logger.go:42: 19:06:09 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.864229 | controller | - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.864233 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.864238 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.864242 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.864246 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.864250 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.864255 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.864259 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.864263 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.864267 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.864272 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.864276 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.864280 | controller | == 1 ]' 2026-03-09 19:10:03.864285 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.864289 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.864293 | controller | - ' else' 2026-03-09 19:10:03.864297 | controller | - ' exit 1' 2026-03-09 19:10:03.864302 | controller | - ' fi' 2026-03-09 19:10:03.864306 | controller | - ' ]' 2026-03-09 19:10:03.864310 | controller | - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.864314 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.864319 | controller | - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.864323 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.864327 | controller | - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.864331 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.864336 | controller | - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.864340 | controller | -1' 2026-03-09 19:10:03.864346 | controller | - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.864350 | controller | -v ''^$''' 2026-03-09 19:10:03.864355 | controller | - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.864359 | controller | - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.864363 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.864367 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.864372 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.864376 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.864380 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.864384 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.864389 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.864393 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.864397 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.864401 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.864406 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.864410 | controller | == 1 ]' 2026-03-09 19:10:03.864414 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.864418 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.864423 | controller | - ' else' 2026-03-09 19:10:03.864427 | controller | - ' exit 1' 2026-03-09 19:10:03.864431 | controller | - ' fi' 2026-03-09 19:10:03.864435 | controller | - ' ]' 2026-03-09 19:10:03.864440 | controller | - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.864446 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.864450 | controller | - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.864454 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.864459 | controller | - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.864463 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.864474 | controller | - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.864479 | controller | -1' 2026-03-09 19:10:03.864483 | controller | - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.864489 | controller | -v ''^$''' 2026-03-09 19:10:03.864493 | controller | - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.864498 | controller | - ' logger.go:42: 19:06:12 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.864502 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.864506 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.864510 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.864515 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.864519 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.864523 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.864527 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.864532 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.864536 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.864542 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.864546 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.864551 | controller | == 1 ]' 2026-03-09 19:10:03.864555 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.864559 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.864563 | controller | - ' else' 2026-03-09 19:10:03.864567 | controller | - ' exit 1' 2026-03-09 19:10:03.864572 | controller | - ' fi' 2026-03-09 19:10:03.864576 | controller | - ' ]' 2026-03-09 19:10:03.864580 | controller | - ' logger.go:42: 19:06:12 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.864585 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.864589 | controller | - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.864593 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.864597 | controller | - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.864602 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.864606 | controller | - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.864610 | controller | -v ''^$''' 2026-03-09 19:10:03.864614 | controller | - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.864619 | controller | -1' 2026-03-09 19:10:03.864623 | controller | - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.864627 | controller | - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.864631 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.864636 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.864640 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.864644 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.864648 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.864653 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.864657 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.864661 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.864665 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.864670 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.864674 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.864678 | controller | == 1 ]' 2026-03-09 19:10:03.864682 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.864687 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.864691 | controller | - ' else' 2026-03-09 19:10:03.864695 | controller | - ' exit 1' 2026-03-09 19:10:03.864700 | controller | - ' fi' 2026-03-09 19:10:03.864704 | controller | - ' ]' 2026-03-09 19:10:03.864708 | controller | - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.864712 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.864717 | controller | - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.864721 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.864727 | controller | - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.864731 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.864735 | controller | - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.864740 | controller | -v ''^$''' 2026-03-09 19:10:03.864750 | controller | - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.864755 | controller | -1' 2026-03-09 19:10:03.864759 | controller | - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-09 19:10:03.864763 | controller | - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.864768 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.864772 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.864776 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.864780 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.864785 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.864789 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.864793 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.864797 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.864802 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.864806 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.864810 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.864815 | controller | == 1 ]' 2026-03-09 19:10:03.864819 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.864823 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.864827 | controller | - ' else' 2026-03-09 19:10:03.864832 | controller | - ' exit 1' 2026-03-09 19:10:03.864836 | controller | - ' fi' 2026-03-09 19:10:03.864840 | controller | - ' ]' 2026-03-09 19:10:03.864844 | controller | - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.864849 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.864853 | controller | - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.864857 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.864861 | controller | - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.864866 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.864870 | controller | - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.864874 | controller | -1' 2026-03-09 19:10:03.864878 | controller | - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.864883 | controller | -v ''^$''' 2026-03-09 19:10:03.864887 | controller | - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.864893 | controller | - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.864897 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.864903 | controller | - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.864907 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.864911 | controller | - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.864918 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.864923 | controller | - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | error: 2026-03-09 19:10:03.864927 | controller | unable to upgrade connection: container not found ("watcher-api")' 2026-03-09 19:10:03.864931 | controller | - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | ++ echo' 2026-03-09 19:10:03.864936 | controller | - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.864940 | controller | 0 == 1 '']''' 2026-03-09 19:10:03.864944 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.864949 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.864953 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.864957 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.864962 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.864966 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.864970 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.864975 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.864979 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.864983 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.864987 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.864992 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.864996 | controller | == 1 ]' 2026-03-09 19:10:03.865007 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.865011 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.865015 | controller | - ' else' 2026-03-09 19:10:03.865020 | controller | - ' exit 1' 2026-03-09 19:10:03.865024 | controller | - ' fi' 2026-03-09 19:10:03.865030 | controller | - ' ]' 2026-03-09 19:10:03.865035 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.865039 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.865043 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.865047 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.865051 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.865056 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.865060 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.865064 | controller | -1' 2026-03-09 19:10:03.865068 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865073 | controller | -v ''^$''' 2026-03-09 19:10:03.865077 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.865081 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865085 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.865090 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865094 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.865098 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.865102 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.865116 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.865120 | controller | ''#'' Global config' 2026-03-09 19:10:03.865124 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865128 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.865133 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865137 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.865141 | controller | - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.865167 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.865194 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.865199 | controller | ''#'' Service config' 2026-03-09 19:10:03.865203 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865207 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.865211 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865216 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.865220 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.865224 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.865228 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.865232 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.865239 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.865243 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.865248 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.865252 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.865256 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.865260 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.865265 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.865269 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.865273 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.865277 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.865283 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.865294 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.865299 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.865303 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.865308 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.865312 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.865318 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.865322 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.865327 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.865331 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.865335 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.865339 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.865343 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.865348 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.865352 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.865356 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.865360 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.865364 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.865368 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.865373 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.865377 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.865381 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865385 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.865390 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865394 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.865399 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.865404 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.865408 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.865412 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.865421 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.865425 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.865429 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.865433 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.865437 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.865442 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.865446 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.865450 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.865454 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.865459 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.865468 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.865473 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.865477 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.865481 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.865487 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.865491 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.865502 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.865507 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.865511 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.865515 | controller | - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865520 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.865524 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.865528 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.865532 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.865537 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.865541 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.865545 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.865549 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.865554 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.865558 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.865562 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.865566 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.865572 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.865577 | controller | == 1 ]' 2026-03-09 19:10:03.865581 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.865585 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.865589 | controller | - ' else' 2026-03-09 19:10:03.865593 | controller | - ' exit 1' 2026-03-09 19:10:03.865598 | controller | - ' fi' 2026-03-09 19:10:03.865602 | controller | - ' ]' 2026-03-09 19:10:03.865606 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.865610 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.865615 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.865619 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.865623 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.865627 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.865631 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.865636 | controller | -1' 2026-03-09 19:10:03.865640 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865646 | controller | -v ''^$''' 2026-03-09 19:10:03.865650 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.865654 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865658 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.865663 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865667 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.865671 | controller | - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.865675 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.865679 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.865684 | controller | ''#'' Global config' 2026-03-09 19:10:03.865688 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865692 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.865696 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865701 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.865705 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.865709 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.865713 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.865717 | controller | ''#'' Service config' 2026-03-09 19:10:03.865721 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865726 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.865730 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865734 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.865738 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.865742 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.865753 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.865758 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.865762 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.865766 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.865772 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.865776 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.865781 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.865785 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.865789 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.865793 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.865797 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.865802 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.865808 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.865812 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.865816 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.865822 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.865826 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.865830 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.865834 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.865839 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.865843 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.865847 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.865851 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.865855 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.865860 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.865864 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.865868 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.865872 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.865876 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.865881 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.865886 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.865891 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.865895 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.865899 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.865903 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.865907 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.865953 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.865957 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.865962 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.865966 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.865970 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.865978 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.865990 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.865994 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.865998 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.866006 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.866011 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.866015 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.866019 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.866023 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.866028 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.866036 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.866040 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.866044 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.866049 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.866053 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.866057 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.866061 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.866066 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.866072 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.866076 | controller | - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866080 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.866085 | controller | - ' logger.go:42: 19:06:21 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.866089 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.866093 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.866098 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.866102 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.866106 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.866110 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.866114 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.866119 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.866123 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.866127 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.866132 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.866136 | controller | == 1 ]' 2026-03-09 19:10:03.866140 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.866156 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.866162 | controller | - ' else' 2026-03-09 19:10:03.866167 | controller | - ' exit 1' 2026-03-09 19:10:03.866171 | controller | - ' fi' 2026-03-09 19:10:03.866175 | controller | - ' ]' 2026-03-09 19:10:03.866207 | controller | - ' logger.go:42: 19:06:21 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.866214 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.866219 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.866225 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.866229 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.866234 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.866238 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.866242 | controller | -1' 2026-03-09 19:10:03.866246 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866250 | controller | -v ''^$''' 2026-03-09 19:10:03.866255 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.866259 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866263 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.866267 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.866280 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.866285 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866289 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.866293 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.866298 | controller | ''#'' Global config' 2026-03-09 19:10:03.866302 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866306 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.866310 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866314 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.866319 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.866323 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.866327 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.866331 | controller | ''#'' Service config' 2026-03-09 19:10:03.866336 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866340 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.866344 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866348 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.866353 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.866357 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.866361 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.866365 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.866370 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.866374 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.866378 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.866382 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.866389 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.866394 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.866398 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.866402 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.866406 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.866410 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.866415 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.866419 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.866423 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.866427 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.866431 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.866436 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.866442 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.866446 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.866450 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.866455 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.866459 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.866463 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.866467 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.866472 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.866482 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.866487 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.866491 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.866495 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.866499 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.866504 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.866508 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.866512 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866516 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.866521 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866525 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.866529 | controller | - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.866533 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.866537 | controller | - ' logger.go:42: 19:06:23 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.866543 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.866552 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.866556 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.866560 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.866565 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.866569 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.866573 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.866577 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.866582 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.866586 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.866590 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.866598 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.866603 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.866607 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.866611 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.866615 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.866620 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.866624 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.866630 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.866634 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.866639 | controller | - ' logger.go:42: 19:06:23 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866643 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.866649 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.866653 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.866657 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.866661 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.866666 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.866670 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.866674 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.866679 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.866683 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.866687 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.866691 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.866703 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.866709 | controller | == 1 ]' 2026-03-09 19:10:03.866713 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.866718 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.866722 | controller | - ' else' 2026-03-09 19:10:03.866726 | controller | - ' exit 1' 2026-03-09 19:10:03.866730 | controller | - ' fi' 2026-03-09 19:10:03.866735 | controller | - ' ]' 2026-03-09 19:10:03.866739 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.866743 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.866747 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.866752 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.866756 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.866760 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.866764 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.866769 | controller | -1' 2026-03-09 19:10:03.866773 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866777 | controller | -v ''^$''' 2026-03-09 19:10:03.866781 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.866786 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866790 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.866794 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866798 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.866803 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.866807 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.866811 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.866815 | controller | ''#'' Global config' 2026-03-09 19:10:03.866820 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866824 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.866828 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866832 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.866837 | controller | - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.866841 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.866845 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.866849 | controller | ''#'' Service config' 2026-03-09 19:10:03.866854 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.866858 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.866862 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.866866 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.866871 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.866875 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.866879 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.866883 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.866889 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.866893 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.866898 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.866902 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.866906 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.866911 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.866915 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.866919 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.866923 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.866934 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.866938 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.866943 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.866947 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.866951 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.866956 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.866960 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.866964 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.866968 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.866973 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.866977 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.866981 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.866987 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.866991 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.866995 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.867000 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.867004 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.867008 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.867012 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.867017 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.867021 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.867025 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.867031 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867035 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.867039 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867044 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.867048 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.867052 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.867056 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.867061 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.867069 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.867073 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.867077 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.867082 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.867086 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.867090 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.867094 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.867099 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.867103 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.867107 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.867115 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.867120 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.867124 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.867135 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.867139 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.867143 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.867167 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.867172 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.867176 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.867180 | controller | - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867184 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.867189 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.867193 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.867197 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.867201 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.867206 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.867212 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.867216 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.867223 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.867227 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.867232 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.867236 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.867240 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.867244 | controller | == 1 ]' 2026-03-09 19:10:03.867249 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.867253 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.867257 | controller | - ' else' 2026-03-09 19:10:03.867261 | controller | - ' exit 1' 2026-03-09 19:10:03.867266 | controller | - ' fi' 2026-03-09 19:10:03.867270 | controller | - ' ]' 2026-03-09 19:10:03.867274 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.867278 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.867283 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.867287 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.867291 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867295 | controller | -v ''^$''' 2026-03-09 19:10:03.867300 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.867304 | controller | -1' 2026-03-09 19:10:03.867308 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.867312 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.867317 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.867321 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867325 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.867329 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867334 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.867338 | controller | - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.867342 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.867346 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.867350 | controller | ''#'' Global config' 2026-03-09 19:10:03.867355 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867359 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.867363 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867367 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.867372 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.867376 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.867380 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.867384 | controller | ''#'' Service config' 2026-03-09 19:10:03.867390 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867395 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.867399 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867413 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.867417 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.867422 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.867426 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.867430 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.867438 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.867442 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.867446 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.867451 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.867455 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.867459 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.867463 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.867467 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.867472 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.867476 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.867480 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.867484 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.867489 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.867493 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.867497 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.867501 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.867505 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.867510 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.867514 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.867518 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.867522 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.867527 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.867531 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.867537 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.867541 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.867545 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.867551 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.867555 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.867560 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.867564 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.867593 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.867598 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867603 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.867607 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867611 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.867616 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.867620 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.867624 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.867635 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.867644 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.867648 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.867653 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.867657 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.867661 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.867665 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.867670 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.867674 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.867678 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.867682 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.867691 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.867695 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.867699 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.867703 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.867708 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.867712 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.867716 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.867720 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.867725 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.867731 | controller | - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867735 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.867739 | controller | - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.867744 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.867748 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.867752 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.867756 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.867761 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.867765 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.867769 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.867775 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.867781 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.867787 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.867793 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.867799 | controller | == 1 ]' 2026-03-09 19:10:03.867804 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.867810 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.867816 | controller | - ' else' 2026-03-09 19:10:03.867822 | controller | - ' exit 1' 2026-03-09 19:10:03.867828 | controller | - ' fi' 2026-03-09 19:10:03.867834 | controller | - ' ]' 2026-03-09 19:10:03.867840 | controller | - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.867846 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.867855 | controller | - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.867862 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.867868 | controller | - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.867874 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.867878 | controller | - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.867882 | controller | -1' 2026-03-09 19:10:03.867887 | controller | - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867891 | controller | -v ''^$''' 2026-03-09 19:10:03.867905 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.867909 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867913 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.867918 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867922 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.867926 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.867931 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.867935 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.867939 | controller | ''#'' Global config' 2026-03-09 19:10:03.867943 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867947 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.867952 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867958 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.867963 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.867967 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.867971 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.867975 | controller | ''#'' Service config' 2026-03-09 19:10:03.867980 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.867984 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.867988 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.867993 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.867998 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.868002 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.868006 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.868012 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.868017 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.868021 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.868025 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.868030 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.868034 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.868038 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.868042 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.868046 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.868051 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.868055 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.868059 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.868063 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.868068 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.868072 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.868076 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.868082 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.868086 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.868090 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.868094 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.868099 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.868105 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.868116 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.868121 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.868125 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.868130 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.868135 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.868140 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.868160 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.868168 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.868172 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.868176 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.868181 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.868185 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.868189 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.868193 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.868198 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.868202 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.868206 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.868210 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.868221 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.868225 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.868229 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.868233 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.868238 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.868242 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.868246 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.868250 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.868255 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.868259 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.868267 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.868271 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.868276 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.868281 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.868286 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.868290 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.868294 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.868298 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.868303 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.868307 | controller | - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.868311 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.868315 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.868320 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.868324 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.868328 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.868332 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.868337 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.868341 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.868345 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.868357 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.868362 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.868366 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.868372 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.868377 | controller | == 1 ]' 2026-03-09 19:10:03.868381 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.868385 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.868389 | controller | - ' else' 2026-03-09 19:10:03.868393 | controller | - ' exit 1' 2026-03-09 19:10:03.868398 | controller | - ' fi' 2026-03-09 19:10:03.868402 | controller | - ' ]' 2026-03-09 19:10:03.868406 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.868410 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.868415 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.868419 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.868423 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.868427 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.868432 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.868436 | controller | -1' 2026-03-09 19:10:03.868440 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.868444 | controller | -v ''^$''' 2026-03-09 19:10:03.868448 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.868454 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.868458 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.868463 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.868470 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.868474 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.868478 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.868483 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.868487 | controller | ''#'' Global config' 2026-03-09 19:10:03.868491 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.868495 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.868499 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.868504 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.868508 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.868512 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.868517 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.868521 | controller | ''#'' Service config' 2026-03-09 19:10:03.868525 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.868529 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.868534 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.868538 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.868542 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.868546 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.868551 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.868555 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.868559 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.868563 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.868567 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.868572 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.868576 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.868580 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.868592 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.868596 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.868600 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.868605 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.868609 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.868613 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.868618 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.868622 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.868628 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.868632 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.868636 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.868641 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.868645 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.868649 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.868653 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.868658 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.868662 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.868666 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.868670 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.868674 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.868679 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.868683 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.868687 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.868691 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.868697 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.868702 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.868706 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.868710 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.868714 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.868719 | controller | - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.868723 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.868727 | controller | - ' logger.go:42: 19:06:32 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.868731 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.868740 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.868744 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.868748 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.868752 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.868757 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.868762 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.868770 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.868774 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.868780 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.868785 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.868800 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.868805 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.868809 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.868813 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.868818 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.868822 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.868826 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.868830 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.868835 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.868839 | controller | - ' logger.go:42: 19:06:32 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.868843 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.868849 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.868855 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.868869 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.868875 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.868880 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.868884 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.868889 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.868893 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.868898 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.868902 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.868906 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.868910 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.868915 | controller | == 1 ]' 2026-03-09 19:10:03.868919 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.868923 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.868928 | controller | - ' else' 2026-03-09 19:10:03.868932 | controller | - ' exit 1' 2026-03-09 19:10:03.868936 | controller | - ' fi' 2026-03-09 19:10:03.868940 | controller | - ' ]' 2026-03-09 19:10:03.868945 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.868949 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.868953 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.868957 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.868961 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.868966 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.868970 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.868976 | controller | -1' 2026-03-09 19:10:03.868982 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.868986 | controller | -v ''^$''' 2026-03-09 19:10:03.868991 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.868995 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.868999 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.869003 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869008 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.869012 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.869016 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.869020 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.869024 | controller | ''#'' Global config' 2026-03-09 19:10:03.869029 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869033 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.869037 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869041 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.869045 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.869050 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.869054 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.869058 | controller | ''#'' Service config' 2026-03-09 19:10:03.869062 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869067 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.869079 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869083 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.869088 | controller | - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.869092 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.869096 | controller | - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.869100 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.869105 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.869109 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.869113 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.869117 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.869122 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.869126 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.869130 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.869134 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.869140 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.869156 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.869163 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.869167 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.869172 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.869176 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.869180 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.869184 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.869188 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.869193 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.869197 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.869201 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.869205 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.869210 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.869214 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.869218 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.869222 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.869226 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.869231 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.869235 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.869239 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.869243 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.869248 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.869252 | controller | - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869256 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.869262 | controller | - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869266 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.869278 | controller | - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.869282 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.869287 | controller | - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.869291 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.869299 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.869304 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.869310 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.869314 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.869318 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.869322 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.869327 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.869331 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.869335 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.869339 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.869348 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.869352 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.869356 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.869360 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.869365 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.869369 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.869373 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.869377 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.869382 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.869386 | controller | - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869390 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.869394 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.869398 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.869403 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.869407 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.869411 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.869415 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.869420 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.869424 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.869428 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.869432 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.869437 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.869441 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.869445 | controller | == 1 ]' 2026-03-09 19:10:03.869451 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.869455 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.869460 | controller | - ' else' 2026-03-09 19:10:03.869464 | controller | - ' exit 1' 2026-03-09 19:10:03.869468 | controller | - ' fi' 2026-03-09 19:10:03.869474 | controller | - ' ]' 2026-03-09 19:10:03.869478 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.869483 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.869487 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.869491 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.869495 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.869500 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.869511 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.869516 | controller | -1' 2026-03-09 19:10:03.869520 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869524 | controller | -v ''^$''' 2026-03-09 19:10:03.869529 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.869533 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869537 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.869543 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869547 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.869551 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.869556 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.869560 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.869564 | controller | ''#'' Global config' 2026-03-09 19:10:03.869568 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869573 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.869577 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869581 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.869585 | controller | - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.869590 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.869594 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.869598 | controller | ''#'' Service config' 2026-03-09 19:10:03.869602 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869606 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.869611 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869615 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.869619 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.869623 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.869628 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.869632 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.869636 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.869640 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.869645 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.869651 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.869655 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.869659 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.869663 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.869668 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.869672 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.869676 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.869680 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.869685 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.869689 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.869693 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.869697 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.869701 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.869706 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.869710 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.869721 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.869726 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.869730 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.869734 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.869738 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.869743 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.869747 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.869751 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.869755 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.869760 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.869764 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.869768 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.869772 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.869776 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869781 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.869785 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.869789 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.869793 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.869802 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.869806 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.869811 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.869819 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.869823 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.869828 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.869832 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.869836 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.869840 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.869844 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.869849 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.869853 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.869857 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.869866 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.869870 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.869874 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.869878 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.869882 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.869887 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.869891 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.869895 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.869899 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.869903 | controller | - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.869908 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.869912 | controller | - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.869916 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.869920 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.869925 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.869935 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.869940 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.869944 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.869948 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.869952 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.869957 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.869961 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.869967 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.869972 | controller | == 1 ]' 2026-03-09 19:10:03.869976 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.869980 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.869984 | controller | - ' else' 2026-03-09 19:10:03.869989 | controller | - ' exit 1' 2026-03-09 19:10:03.869993 | controller | - ' fi' 2026-03-09 19:10:03.869999 | controller | - ' ]' 2026-03-09 19:10:03.870003 | controller | - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.870008 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.870012 | controller | - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.870016 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.870020 | controller | - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.870025 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.870029 | controller | - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.870033 | controller | -1' 2026-03-09 19:10:03.870037 | controller | - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870041 | controller | -v ''^$''' 2026-03-09 19:10:03.870046 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.870050 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870054 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.870058 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870063 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.870067 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.870071 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.870084 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.870088 | controller | ''#'' Global config' 2026-03-09 19:10:03.870093 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870097 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.870101 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870105 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.870109 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.870114 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.870118 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.870122 | controller | ''#'' Service config' 2026-03-09 19:10:03.870126 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870131 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.870135 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870139 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.870155 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.870161 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.870168 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.870172 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.870176 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.870180 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.870185 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.870198 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.870203 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.870208 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.870212 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.870216 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.870220 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.870224 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.870229 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.870233 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.870237 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.870241 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.870245 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.870251 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.870256 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.870260 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.870264 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.870268 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.870273 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.870277 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.870281 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.870285 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.870290 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.870294 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.870298 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.870302 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.870306 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.870313 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.870317 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.870321 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870325 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.870330 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870334 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.870338 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.870342 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.870347 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.870351 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.870361 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.870365 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.870369 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.870374 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.870378 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.870382 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.870386 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.870390 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.870402 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.870406 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.870415 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.870419 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.870423 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.870427 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.870432 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.870436 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.870440 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.870444 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.870448 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.870453 | controller | - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870457 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.870461 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.870465 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.870470 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.870474 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.870480 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.870484 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.870488 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.870492 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.870497 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.870501 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.870505 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.870509 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.870514 | controller | == 1 ]' 2026-03-09 19:10:03.870518 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.870522 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.870526 | controller | - ' else' 2026-03-09 19:10:03.870531 | controller | - ' exit 1' 2026-03-09 19:10:03.870535 | controller | - ' fi' 2026-03-09 19:10:03.870539 | controller | - ' ]' 2026-03-09 19:10:03.870543 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.870548 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.870552 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.870556 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.870560 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.870565 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.870569 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.870573 | controller | -1' 2026-03-09 19:10:03.870577 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870582 | controller | -v ''^$''' 2026-03-09 19:10:03.870586 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.870590 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870594 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.870599 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870603 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.870607 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.870611 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.870615 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.870620 | controller | ''#'' Global config' 2026-03-09 19:10:03.870624 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870628 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.870634 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870638 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.870643 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.870654 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.870658 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.870663 | controller | ''#'' Service config' 2026-03-09 19:10:03.870669 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870673 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.870677 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870682 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.870687 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.870691 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.870695 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.870700 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.870704 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.870708 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.870712 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.870717 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.870721 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.870725 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.870729 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.870734 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.870738 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.870742 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.870746 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.870751 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.870755 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.870759 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.870763 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.870767 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.870772 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.870776 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.870780 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.870784 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.870790 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.870794 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.870799 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.870804 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.870809 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.870813 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.870817 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.870821 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.870825 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.870830 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.870840 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.870845 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870849 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.870854 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.870858 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.870862 | controller | - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.870866 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.870871 | controller | - ' logger.go:42: 19:06:41 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.870875 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.870883 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.870890 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.870895 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.870899 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.870903 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.870907 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.870912 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.870916 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.870920 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.870924 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.870933 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.870937 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.870941 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.870945 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.870949 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.870954 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.870958 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.870964 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.870968 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.870972 | controller | - ' logger.go:42: 19:06:41 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.870976 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.870981 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.870985 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.870989 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.870993 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.870998 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.871002 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.871006 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.871010 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.871015 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.871019 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.871023 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.871027 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.871032 | controller | == 1 ]' 2026-03-09 19:10:03.871036 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.871040 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.871044 | controller | - ' else' 2026-03-09 19:10:03.871048 | controller | - ' exit 1' 2026-03-09 19:10:03.871053 | controller | - ' fi' 2026-03-09 19:10:03.871057 | controller | - ' ]' 2026-03-09 19:10:03.871061 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.871065 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.871070 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.871081 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.871085 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.871089 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.871094 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.871098 | controller | -1' 2026-03-09 19:10:03.871102 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871106 | controller | -v ''^$''' 2026-03-09 19:10:03.871111 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.871115 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.871119 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.871123 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871128 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.871132 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.871136 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.871140 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.871164 | controller | ''#'' Global config' 2026-03-09 19:10:03.871175 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.871179 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.871183 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871188 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.871192 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.871196 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.871202 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.871206 | controller | ''#'' Service config' 2026-03-09 19:10:03.871211 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.871215 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.871219 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871223 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.871228 | controller | - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.871232 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.871236 | controller | - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.871240 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.871245 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.871249 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.871253 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.871257 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.871262 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.871266 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.871270 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.871274 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.871278 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.871283 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.871287 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.871291 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.871295 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.871299 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.871304 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.871308 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.871320 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.871324 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.871330 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.871335 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.871339 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.871343 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.871348 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.871352 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.871356 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.871360 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.871364 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.871369 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.871373 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.871377 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.871381 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.871386 | controller | - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.871390 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.871394 | controller | - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871398 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.871403 | controller | - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.871407 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.871411 | controller | - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.871415 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.871424 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.871428 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.871432 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.871438 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.871442 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.871446 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.871451 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.871455 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.871459 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.871465 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.871473 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.871479 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.871484 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.871488 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.871492 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.871496 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.871501 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.871505 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.871509 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.871513 | controller | - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.871518 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.871529 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.871534 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.871538 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.871542 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.871547 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.871551 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.871557 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.871627 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.871635 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.871640 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.871646 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.871652 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.871658 | controller | == 1 ]' 2026-03-09 19:10:03.871664 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.871669 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.871675 | controller | - ' else' 2026-03-09 19:10:03.871680 | controller | - ' exit 1' 2026-03-09 19:10:03.871686 | controller | - ' fi' 2026-03-09 19:10:03.871692 | controller | - ' ]' 2026-03-09 19:10:03.871697 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.871701 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.871705 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.871710 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.871714 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.871718 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.871722 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871726 | controller | -v ''^$''' 2026-03-09 19:10:03.871731 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.871735 | controller | -1' 2026-03-09 19:10:03.871739 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.871745 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.871755 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.871761 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871767 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.871773 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.871778 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.871784 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.871788 | controller | ''#'' Global config' 2026-03-09 19:10:03.871792 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.871797 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.871801 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871805 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.871809 | controller | - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.871813 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.871818 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.871822 | controller | ''#'' Service config' 2026-03-09 19:10:03.871826 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.871830 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.871838 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.871843 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.871847 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.871851 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.871855 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.871859 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.871864 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.871878 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.871883 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.871888 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.871892 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.871896 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.871901 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.871905 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.871909 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.871913 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.871917 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.871922 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.871928 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.871932 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.871937 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.871941 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.871945 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.871949 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.871954 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.871958 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.871962 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.871966 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.871971 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.871975 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.871979 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.871983 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.871987 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.871992 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.871996 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.872000 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.872004 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.872009 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872013 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.872017 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872021 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.872025 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.872030 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.872034 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.872038 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.872047 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.872051 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.872055 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.872059 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.872063 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.872076 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.872083 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.872087 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.872091 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.872096 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.872104 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.872108 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.872113 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.872117 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.872121 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.872125 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.872130 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.872134 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.872138 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.872142 | controller | - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872159 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.872166 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.872171 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.872175 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.872179 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.872185 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.872191 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.872197 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.872203 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.872209 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.872215 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.872221 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.872226 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.872230 | controller | == 1 ]' 2026-03-09 19:10:03.872235 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.872239 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.872243 | controller | - ' else' 2026-03-09 19:10:03.872248 | controller | - ' exit 1' 2026-03-09 19:10:03.872252 | controller | - ' fi' 2026-03-09 19:10:03.872256 | controller | - ' ]' 2026-03-09 19:10:03.872260 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.872265 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.872269 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.872273 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.872278 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.872284 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.872288 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.872293 | controller | -1' 2026-03-09 19:10:03.872297 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872301 | controller | -v ''^$''' 2026-03-09 19:10:03.872306 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.872310 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872314 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.872318 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872322 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.872327 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.872333 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.872337 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.872342 | controller | ''#'' Global config' 2026-03-09 19:10:03.872346 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872359 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.872364 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872369 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.872373 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.872377 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.872381 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.872386 | controller | ''#'' Service config' 2026-03-09 19:10:03.872393 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872398 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.872402 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872406 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.872410 | controller | - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.872417 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.872421 | controller | - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.872426 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.872431 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.872435 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.872439 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.872444 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.872448 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.872452 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.872458 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.872463 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.872467 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.872471 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.872476 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.872480 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.872484 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.872488 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.872492 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.872497 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.872501 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.872505 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.872509 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.872514 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.872518 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.872522 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.872526 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.872531 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.872535 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.872539 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.872543 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.872548 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.872559 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.872564 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.872568 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.872572 | controller | - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872577 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.872581 | controller | - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872585 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.872589 | controller | - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.872594 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.872598 | controller | - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.872602 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.872612 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.872617 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.872621 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.872625 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.872629 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.872633 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.872638 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.872642 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.872646 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.872650 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.872661 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.872665 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.872669 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.872673 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.872677 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.872682 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.872686 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.872690 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.872694 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.872699 | controller | - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872703 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.872707 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.872713 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.872717 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.872722 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.872727 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.872732 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.872738 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.872743 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.872749 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.872756 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.872762 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.872768 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.872773 | controller | == 1 ]' 2026-03-09 19:10:03.872777 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.872783 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.872787 | controller | - ' else' 2026-03-09 19:10:03.872792 | controller | - ' exit 1' 2026-03-09 19:10:03.872803 | controller | - ' fi' 2026-03-09 19:10:03.872808 | controller | - ' ]' 2026-03-09 19:10:03.872812 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.872819 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.872823 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.872828 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.872832 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.872836 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.872840 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.872845 | controller | -1' 2026-03-09 19:10:03.872849 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872853 | controller | -v ''^$''' 2026-03-09 19:10:03.872858 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.872862 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872866 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.872870 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872875 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.872879 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.872883 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.872887 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.872892 | controller | ''#'' Global config' 2026-03-09 19:10:03.872896 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872900 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.872905 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872909 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.872913 | controller | - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.872917 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.872922 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.872926 | controller | ''#'' Service config' 2026-03-09 19:10:03.872930 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.872934 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.872939 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.872943 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.872947 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.872951 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.872956 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.872960 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.872964 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.872970 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.872974 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.872979 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.872983 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.872987 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.872991 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.872997 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.873003 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.873007 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.873011 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.873015 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.873020 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.873031 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.873036 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.873042 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.873048 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.873054 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.873059 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.873065 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.873070 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.873076 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.873082 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.873087 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.873093 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.873099 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.873105 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.873110 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.873114 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.873119 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.873123 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.873127 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.873131 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.873135 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.873142 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.873162 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.873166 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.873171 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.873175 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.873184 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.873189 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.873200 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.873212 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.873218 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.873224 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.873230 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.873236 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.873241 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.873247 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.873258 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.873269 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.873275 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.873281 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.873286 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.873292 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.873297 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.873316 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.873324 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.873329 | controller | - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.873333 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.873337 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.873341 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.873346 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.873350 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.873354 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.873358 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.873363 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.873367 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.873371 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.873378 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.873383 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.873387 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.873391 | controller | == 1 ]' 2026-03-09 19:10:03.873396 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.873400 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.873404 | controller | - ' else' 2026-03-09 19:10:03.873408 | controller | - ' exit 1' 2026-03-09 19:10:03.873413 | controller | - ' fi' 2026-03-09 19:10:03.873417 | controller | - ' ]' 2026-03-09 19:10:03.873421 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.873426 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.873430 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.873434 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.873438 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.873443 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.873447 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.873451 | controller | -1' 2026-03-09 19:10:03.873455 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.873460 | controller | -v ''^$''' 2026-03-09 19:10:03.873464 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.873468 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.873472 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.873477 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.873481 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.873485 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.873489 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.873494 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.873498 | controller | ''#'' Global config' 2026-03-09 19:10:03.873502 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.873506 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.873511 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.873515 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.873519 | controller | - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.873523 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.873528 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.873532 | controller | ''#'' Service config' 2026-03-09 19:10:03.873536 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.873540 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.873545 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.873549 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.873555 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.873559 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.873563 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.873575 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.873583 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.873587 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.873591 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.873596 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.873600 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.873604 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.873609 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.873613 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.873617 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.873621 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.873625 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.873630 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.873634 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.873638 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.873642 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.873646 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.873651 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.873655 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.873659 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.873663 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.873667 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.873672 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.873676 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.873680 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.873684 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.873689 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.873693 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.873699 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.873704 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.873709 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.873713 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.873717 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.873721 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.873726 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.873730 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.873734 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.873738 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.873744 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.873748 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.873757 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.873761 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.873772 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.873776 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.873780 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.873785 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.873789 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.873793 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.873798 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.873802 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.873810 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.873814 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.873819 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.873825 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.873829 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.873833 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.873838 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.873842 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.873846 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.873850 | controller | - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.873855 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.873859 | controller | - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.873865 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.873870 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.873874 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.873878 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.873882 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.873887 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.873891 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.873895 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.873899 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.873904 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.873909 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.873914 | controller | == 1 ]' 2026-03-09 19:10:03.873918 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.873922 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.873927 | controller | - ' else' 2026-03-09 19:10:03.873931 | controller | - ' exit 1' 2026-03-09 19:10:03.873935 | controller | - ' fi' 2026-03-09 19:10:03.873939 | controller | - ' ]' 2026-03-09 19:10:03.873944 | controller | - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.873948 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.873952 | controller | - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.873956 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.873961 | controller | - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.873965 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.873969 | controller | - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.873973 | controller | -1' 2026-03-09 19:10:03.873978 | controller | - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.873982 | controller | -v ''^$''' 2026-03-09 19:10:03.873986 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.873990 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.873994 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.873999 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874003 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.874007 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.874018 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.874022 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.874026 | controller | ''#'' Global config' 2026-03-09 19:10:03.874031 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874035 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.874039 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874044 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.874048 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.874052 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.874058 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.874062 | controller | ''#'' Service config' 2026-03-09 19:10:03.874067 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874071 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.874075 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874079 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.874083 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.874088 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.874092 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.874096 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.874101 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.874105 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.874109 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.874113 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.874117 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.874122 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.874126 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.874130 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.874134 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.874138 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.874143 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.874167 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.874174 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.874178 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.874182 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.874186 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.874191 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.874195 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.874199 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.874203 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.874208 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.874214 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.874218 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.874222 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.874226 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.874238 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.874242 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.874247 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.874251 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.874255 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.874260 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.874264 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874268 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.874272 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874277 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.874281 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.874285 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.874289 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.874293 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.874302 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.874306 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.874311 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.874315 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.874319 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.874323 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.874327 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.874332 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.874336 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.874340 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.874349 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.874353 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.874357 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.874361 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.874366 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.874373 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.874377 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.874383 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.874387 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.874391 | controller | - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874396 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.874400 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.874404 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.874409 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.874413 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.874417 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.874421 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.874426 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.874430 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.874434 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.874438 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.874443 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.874454 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.874458 | controller | == 1 ]' 2026-03-09 19:10:03.874463 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.874467 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.874471 | controller | - ' else' 2026-03-09 19:10:03.874476 | controller | - ' exit 1' 2026-03-09 19:10:03.874480 | controller | - ' fi' 2026-03-09 19:10:03.874484 | controller | - ' ]' 2026-03-09 19:10:03.874488 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.874493 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.874497 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.874501 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.874505 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.874510 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.874514 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.874518 | controller | -1' 2026-03-09 19:10:03.874522 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874526 | controller | -v ''^$''' 2026-03-09 19:10:03.874531 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.874535 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874539 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.874543 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874548 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.874552 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.874556 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.874564 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.874568 | controller | ''#'' Global config' 2026-03-09 19:10:03.874572 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874577 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.874581 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874585 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.874589 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.874594 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.874598 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.874602 | controller | ''#'' Service config' 2026-03-09 19:10:03.874606 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874611 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.874615 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874619 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.874623 | controller | - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.874628 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.874632 | controller | - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.874636 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.874640 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.874645 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.874649 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.874653 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.874657 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.874662 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.874666 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.874670 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.874674 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.874678 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.874689 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.874693 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.874698 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.874702 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.874708 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.874712 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.874719 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.874723 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.874728 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.874732 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.874738 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.874742 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.874746 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.874750 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.874755 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.874759 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.874763 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.874767 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.874772 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.874776 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.874780 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.874784 | controller | - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874788 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.874793 | controller | - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.874797 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.874801 | controller | - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.874805 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.874810 | controller | - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.874814 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.874822 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.874827 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.874831 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.874835 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.874839 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.874843 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.874847 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.874852 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.874856 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.874860 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.874870 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.874874 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.874879 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.874883 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.874894 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.874898 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.874903 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.874907 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.874911 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.874917 | controller | - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.874921 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.874926 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.874930 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.874934 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.874938 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.874943 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.874947 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.874951 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.874955 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.874960 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.874964 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.874968 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.874972 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.874977 | controller | == 1 ]' 2026-03-09 19:10:03.874981 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.874985 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.874989 | controller | - ' else' 2026-03-09 19:10:03.874994 | controller | - ' exit 1' 2026-03-09 19:10:03.874998 | controller | - ' fi' 2026-03-09 19:10:03.875004 | controller | - ' ]' 2026-03-09 19:10:03.875008 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.875012 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.875017 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.875021 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.875025 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.875030 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.875034 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.875038 | controller | -1' 2026-03-09 19:10:03.875042 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875056 | controller | -v ''^$''' 2026-03-09 19:10:03.875060 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.875065 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875069 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.875073 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875078 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.875082 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.875086 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.875090 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.875095 | controller | ''#'' Global config' 2026-03-09 19:10:03.875099 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875103 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.875107 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875111 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.875116 | controller | - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.875120 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.875124 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.875128 | controller | ''#'' Service config' 2026-03-09 19:10:03.875133 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875137 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.875141 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875165 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.875180 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.875184 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.875189 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.875193 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.875199 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.875204 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.875208 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.875212 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.875216 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.875221 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.875225 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.875229 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.875233 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.875238 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.875244 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.875248 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.875252 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.875257 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.875261 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.875265 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.875269 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.875274 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.875279 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.875284 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.875288 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.875292 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.875296 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.875301 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.875305 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.875309 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.875313 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.875317 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.875322 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.875326 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.875330 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.875334 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875339 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.875343 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875347 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.875351 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.875356 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.875360 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.875364 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.875380 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.875384 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.875388 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.875393 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.875398 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.875403 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.875407 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.875411 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.875415 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.875420 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.875428 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.875432 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.875437 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.875441 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.875445 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.875449 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.875454 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.875458 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.875462 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.875466 | controller | - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875471 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.875475 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.875479 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.875483 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.875489 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.875493 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.875498 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.875502 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.875506 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.875510 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.875515 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.875519 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.875523 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.875528 | controller | == 1 ]' 2026-03-09 19:10:03.875532 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.875536 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.875540 | controller | - ' else' 2026-03-09 19:10:03.875545 | controller | - ' exit 1' 2026-03-09 19:10:03.875549 | controller | - ' fi' 2026-03-09 19:10:03.875553 | controller | - ' ]' 2026-03-09 19:10:03.875572 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.875579 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.875587 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.875592 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.875596 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.875600 | controller | -1' 2026-03-09 19:10:03.875604 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875609 | controller | -v ''^$''' 2026-03-09 19:10:03.875613 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.875617 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.875621 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.875625 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875637 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.875641 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875646 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.875650 | controller | - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.875654 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.875658 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.875663 | controller | ''#'' Global config' 2026-03-09 19:10:03.875667 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875671 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.875675 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875680 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.875684 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.875688 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.875692 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.875697 | controller | ''#'' Service config' 2026-03-09 19:10:03.875701 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875705 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.875710 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875714 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.875718 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.875722 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.875726 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.875731 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.875735 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.875739 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.875747 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.875751 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.875758 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.875762 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.875766 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.875771 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.875775 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.875779 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.875783 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.875788 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.875792 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.875796 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.875800 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.875806 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.875810 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.875815 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.875819 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.875823 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.875828 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.875832 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.875844 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.875849 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.875853 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.875857 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.875862 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.875866 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.875870 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.875875 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.875879 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.875883 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.875887 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.875891 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.875896 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.875900 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.875904 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.875908 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.875914 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.875923 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.875927 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.875931 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.875936 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.875940 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.875944 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.875948 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.875953 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.875957 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.875961 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.875970 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.875974 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.875978 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.875982 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.875986 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.875991 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.875995 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.875999 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.876003 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.876008 | controller | - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876012 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.876016 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.876020 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.876025 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.876029 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.876033 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.876038 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.876042 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.876046 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.876050 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.876061 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.876066 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.876070 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.876076 | controller | == 1 ]' 2026-03-09 19:10:03.876080 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.876085 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.876089 | controller | - ' else' 2026-03-09 19:10:03.876093 | controller | - ' exit 1' 2026-03-09 19:10:03.876098 | controller | - ' fi' 2026-03-09 19:10:03.876102 | controller | - ' ]' 2026-03-09 19:10:03.876106 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.876111 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.876117 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.876121 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.876125 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.876130 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.876134 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.876138 | controller | -1' 2026-03-09 19:10:03.876142 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876163 | controller | -v ''^$''' 2026-03-09 19:10:03.876167 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.876172 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876176 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.876180 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876184 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.876189 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.876193 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.876197 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.876201 | controller | ''#'' Global config' 2026-03-09 19:10:03.876206 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876210 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.876214 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876218 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.876223 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.876227 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.876231 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.876235 | controller | ''#'' Service config' 2026-03-09 19:10:03.876240 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876244 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.876248 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876252 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.876257 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.876261 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.876265 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.876269 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.876276 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.876280 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.876284 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.876289 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.876293 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.876297 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.876301 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.876313 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.876318 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.876322 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.876327 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.876331 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.876335 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.876339 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.876344 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.876348 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.876352 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.876356 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.876361 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.876365 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.876369 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.876373 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.876378 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.876382 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.876388 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.876392 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.876396 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.876401 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.876405 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.876409 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.876414 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.876419 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876424 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.876428 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876432 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.876436 | controller | - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.876441 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.876445 | controller | - ' logger.go:42: 19:07:04 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.876449 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.876458 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.876462 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.876466 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.876470 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.876474 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.876479 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.876485 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.876489 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.876493 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.876497 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.876506 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.876517 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.876521 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.876525 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.876529 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.876534 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.876538 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.876542 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.876547 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.876551 | controller | - ' logger.go:42: 19:07:04 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876555 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.876559 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.876564 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.876568 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.876572 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.876576 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.876581 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.876587 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.876591 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.876597 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.876601 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.876606 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.876610 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.876615 | controller | == 1 ]' 2026-03-09 19:10:03.876619 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.876623 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.876628 | controller | - ' else' 2026-03-09 19:10:03.876632 | controller | - ' exit 1' 2026-03-09 19:10:03.876636 | controller | - ' fi' 2026-03-09 19:10:03.876640 | controller | - ' ]' 2026-03-09 19:10:03.876645 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.876649 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.876653 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.876658 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.876664 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.876668 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.876672 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.876677 | controller | -1' 2026-03-09 19:10:03.876681 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876685 | controller | -v ''^$''' 2026-03-09 19:10:03.876689 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.876694 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876698 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.876702 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876706 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.876711 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.876715 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.876719 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.876723 | controller | ''#'' Global config' 2026-03-09 19:10:03.876728 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876732 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.876736 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876740 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.876745 | controller | - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.876749 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.876753 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.876757 | controller | ''#'' Service config' 2026-03-09 19:10:03.876762 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876766 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.876770 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876783 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.876788 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.876792 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.876797 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.876804 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.876808 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.876813 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.876817 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.876821 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.876826 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.876830 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.876834 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.876838 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.876843 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.876847 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.876851 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.876855 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.876859 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.876864 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.876868 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.876872 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.876876 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.876881 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.876885 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.876889 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.876894 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.876898 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.876902 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.876906 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.876910 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.876916 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.876921 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.876926 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.876931 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.876935 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.876939 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.876943 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.876948 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.876952 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.876956 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.876961 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.876972 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.876976 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.876981 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.876989 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.876994 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.876998 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.877002 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.877006 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.877011 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.877015 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.877019 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.877023 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.877028 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.877036 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.877040 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.877045 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.877049 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.877053 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.877057 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.877062 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.877066 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.877070 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.877076 | controller | - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.877081 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.877085 | controller | - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | running 2026-03-09 19:10:03.877089 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.877094 | controller | - ' oc project watcher-kuttl-default' 2026-03-09 19:10:03.877098 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-09 19:10:03.877102 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-09 19:10:03.877106 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-09 19:10:03.877111 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-09 19:10:03.877115 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-09 19:10:03.877119 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-09 19:10:03.877123 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-09 19:10:03.877128 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-09 19:10:03.877132 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-09 19:10:03.877136 | controller | == 1 ]' 2026-03-09 19:10:03.877142 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-09 19:10:03.877159 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-09 19:10:03.877164 | controller | - ' else' 2026-03-09 19:10:03.877168 | controller | - ' exit 1' 2026-03-09 19:10:03.877172 | controller | - ' fi' 2026-03-09 19:10:03.877177 | controller | - ' ]' 2026-03-09 19:10:03.877181 | controller | - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-09 19:10:03.877185 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.877190 | controller | - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | Already 2026-03-09 19:10:03.877194 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-09 19:10:03.877198 | controller | - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-09 19:10:03.877203 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-09 19:10:03.877209 | controller | - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-09 19:10:03.877213 | controller | -1' 2026-03-09 19:10:03.877225 | controller | - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.877229 | controller | -v ''^$''' 2026-03-09 19:10:03.877234 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-09 19:10:03.877238 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.877242 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-09 19:10:03.877247 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.877251 | controller | -c ''^# Global config''' 2026-03-09 19:10:03.877255 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.877259 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-09 19:10:03.877264 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.877268 | controller | ''#'' Global config' 2026-03-09 19:10:03.877272 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.877276 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.877281 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.877287 | controller | -c ''^# Service config''' 2026-03-09 19:10:03.877291 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.877296 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-09 19:10:03.877300 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.877304 | controller | ''#'' Service config' 2026-03-09 19:10:03.877310 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.877315 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.877319 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.877323 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-09 19:10:03.877327 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.877332 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-09 19:10:03.877336 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.877340 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_TMF4TY42kuJMsea81ze:13vmHCdv7i6xfnJl_U0Ztc4nS_iYSf4d@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-09 19:10:03.877345 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-09 19:10:03.877349 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-09 19:10:03.877353 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-09 19:10:03.877357 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-09 19:10:03.877362 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-09 19:10:03.877366 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.877370 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-09 19:10:03.877374 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-09 19:10:03.877379 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-09 19:10:03.877383 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.877387 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.877391 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-09 19:10:03.877396 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-09 19:10:03.877400 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-09 19:10:03.877404 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-09 19:10:03.877408 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-09 19:10:03.877412 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-09 19:10:03.877417 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-09 19:10:03.877421 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-09 19:10:03.877436 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-09 19:10:03.877441 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-09 19:10:03.877445 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-09 19:10:03.877450 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-09 19:10:03.877454 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-09 19:10:03.877458 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-09 19:10:03.877463 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-09 19:10:03.877467 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-09 19:10:03.877471 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-09 19:10:03.877476 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-09 19:10:03.877480 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-09 19:10:03.877484 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-09 19:10:03.877489 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.877494 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.877499 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-09 19:10:03.877503 | controller | -czPo ''TimeOut 80''' 2026-03-09 19:10:03.877507 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-09 19:10:03.877512 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-09 19:10:03.877516 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-09 19:10:03.877520 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.877529 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.877533 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.877537 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.877543 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.877547 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.877552 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.877556 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.877560 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-09 19:10:03.877564 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-09 19:10:03.877569 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-09 19:10:03.877577 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-09 19:10:03.877581 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-09 19:10:03.877586 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-09 19:10:03.877591 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-09 19:10:03.877596 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-09 19:10:03.877600 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-09 19:10:03.877604 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-09 19:10:03.877608 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-09 19:10:03.877613 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-09 19:10:03.877617 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-09 19:10:03.877621 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.877626 | controller | - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | test step 2026-03-09 19:10:03.877630 | controller | completed 4-deploy-with-precreated-account' 2026-03-09 19:10:03.877634 | controller | - ' logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher' 2026-03-09 19:10:03.877639 | controller | - ' logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | running command: [sh -c 2026-03-09 19:10:03.877643 | controller | set -ex' 2026-03-09 19:10:03.877653 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.877658 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.877662 | controller | - ' ]' 2026-03-09 19:10:03.877667 | controller | - ' logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.877671 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.877676 | controller | - ' logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.877680 | controller | - ' logger.go:42: 19:07:11 | watcher/5-cleanup-watcher | + ''['' 1 == 0 '']''' 2026-03-09 19:10:03.877684 | controller | - ' logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | running command: [sh -c 2026-03-09 19:10:03.877688 | controller | set -ex' 2026-03-09 19:10:03.877693 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.877697 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.877701 | controller | - ' ]' 2026-03-09 19:10:03.877706 | controller | - ' logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-09 19:10:03.877710 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-09 19:10:03.877714 | controller | - ' logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-09 19:10:03.877718 | controller | - ' logger.go:42: 19:07:16 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.877723 | controller | - ' logger.go:42: 19:07:16 | watcher/5-cleanup-watcher | test step completed 5-cleanup-watcher' 2026-03-09 19:10:03.877727 | controller | - ' logger.go:42: 19:07:16 | watcher | skipping kubernetes event logging' 2026-03-09 19:10:03.877731 | controller | - === CONT kuttl/harness/watcher-cinder 2026-03-09 19:10:03.877736 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder | Skipping creation of user-supplied 2026-03-09 19:10:03.877742 | controller | namespace: watcher-kuttl-default' 2026-03-09 19:10:03.877746 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/0-cleanup-watcher | starting test step 2026-03-09 19:10:03.877752 | controller | 0-cleanup-watcher' 2026-03-09 19:10:03.877764 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/0-cleanup-watcher | test step completed 2026-03-09 19:10:03.877776 | controller | 0-cleanup-watcher' 2026-03-09 19:10:03.877783 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | starting 2026-03-09 19:10:03.877788 | controller | test step 1-deploy-watcher-no-cinder' 2026-03-09 19:10:03.877792 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.877800 | controller | created' 2026-03-09 19:10:03.877804 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.877808 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.877813 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.877817 | controller | service' 2026-03-09 19:10:03.877821 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.877826 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.877830 | controller | - ' ]' 2026-03-09 19:10:03.877834 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.877838 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.877843 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.877851 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.877855 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.877860 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.877864 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.877868 | controller | - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.877872 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.877877 | controller | - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.877881 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.877885 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.877889 | controller | service' 2026-03-09 19:10:03.877893 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.877898 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.877902 | controller | - ' ]' 2026-03-09 19:10:03.877906 | controller | - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.877910 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.877914 | controller | - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.877919 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.877923 | controller | - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.877927 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.877931 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.877936 | controller | - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.877940 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.877955 | controller | - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.877960 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.877964 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.877968 | controller | service' 2026-03-09 19:10:03.877973 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.877977 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.877981 | controller | - ' ]' 2026-03-09 19:10:03.877985 | controller | - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.877990 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.877994 | controller | - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878000 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878005 | controller | - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.878009 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.878013 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.878017 | controller | - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878021 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878026 | controller | - ' logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878030 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878034 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878038 | controller | service' 2026-03-09 19:10:03.878044 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878049 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878053 | controller | - ' ]' 2026-03-09 19:10:03.878057 | controller | - ' logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878061 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878065 | controller | - ' logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878070 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878074 | controller | - ' logger.go:42: 19:07:20 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.878078 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.878082 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.878087 | controller | - ' logger.go:42: 19:07:20 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878091 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878095 | controller | - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878099 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878104 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878108 | controller | service' 2026-03-09 19:10:03.878112 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878116 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878121 | controller | - ' ]' 2026-03-09 19:10:03.878125 | controller | - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878129 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878133 | controller | - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878137 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878142 | controller | - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.878159 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.878167 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.878171 | controller | - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878175 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878179 | controller | - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878184 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878188 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878192 | controller | service' 2026-03-09 19:10:03.878196 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878202 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878207 | controller | - ' ]' 2026-03-09 19:10:03.878211 | controller | - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878215 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878219 | controller | - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878224 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878228 | controller | - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.878240 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.878245 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.878249 | controller | - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878253 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878258 | controller | - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878262 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878266 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878270 | controller | service' 2026-03-09 19:10:03.878275 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878279 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878283 | controller | - ' ]' 2026-03-09 19:10:03.878287 | controller | - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878292 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878296 | controller | - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878300 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878304 | controller | - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.878309 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.878313 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.878317 | controller | - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878321 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878326 | controller | - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878330 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878336 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878340 | controller | service' 2026-03-09 19:10:03.878345 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878349 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878353 | controller | - ' ]' 2026-03-09 19:10:03.878357 | controller | - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878362 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878366 | controller | - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878370 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878374 | controller | - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.878379 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.878383 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.878387 | controller | - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878393 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878397 | controller | - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878401 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878406 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878410 | controller | service' 2026-03-09 19:10:03.878414 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878418 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878422 | controller | - ' ]' 2026-03-09 19:10:03.878427 | controller | - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878431 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878435 | controller | - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878439 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878444 | controller | - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.878448 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.878452 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.878456 | controller | - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878461 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878465 | controller | - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878469 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878473 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878478 | controller | service' 2026-03-09 19:10:03.878482 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878486 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878490 | controller | - ' ]' 2026-03-09 19:10:03.878495 | controller | - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878499 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878503 | controller | - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878515 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878519 | controller | - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-09 19:10:03.878524 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-09 19:10:03.878528 | controller | namespace "watcher-kuttl-default"' 2026-03-09 19:10:03.878532 | controller | - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878536 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878541 | controller | - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878545 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878549 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878553 | controller | service' 2026-03-09 19:10:03.878557 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878562 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878566 | controller | - ' ]' 2026-03-09 19:10:03.878570 | controller | - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878574 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878579 | controller | - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878584 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878589 | controller | - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | Error 2026-03-09 19:10:03.878593 | controller | from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-09 19:10:03.878597 | controller | is waiting to start: ContainerCreating' 2026-03-09 19:10:03.878601 | controller | - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878606 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878610 | controller | - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878614 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878620 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878624 | controller | service' 2026-03-09 19:10:03.878628 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878633 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878637 | controller | - ' ]' 2026-03-09 19:10:03.878641 | controller | - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878645 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878649 | controller | - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878654 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878658 | controller | - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | Error 2026-03-09 19:10:03.878662 | controller | from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-09 19:10:03.878666 | controller | is waiting to start: ContainerCreating' 2026-03-09 19:10:03.878671 | controller | - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878675 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878679 | controller | - ' logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878683 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878687 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878692 | controller | service' 2026-03-09 19:10:03.878696 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878700 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878704 | controller | - ' ]' 2026-03-09 19:10:03.878709 | controller | - ' logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878713 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878717 | controller | - ' logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878723 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878727 | controller | - ' logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878731 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878736 | controller | - ' logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878740 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878746 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878752 | controller | service' 2026-03-09 19:10:03.878758 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878764 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878770 | controller | - ' ]' 2026-03-09 19:10:03.878775 | controller | - ' logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878782 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878786 | controller | - ' logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878798 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878803 | controller | - ' logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878807 | controller | 0 == 2 '']''' 2026-03-09 19:10:03.878811 | controller | - ' logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878816 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878820 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878824 | controller | service' 2026-03-09 19:10:03.878829 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878833 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878837 | controller | - ' ]' 2026-03-09 19:10:03.878842 | controller | - ' logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878846 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878850 | controller | - ' logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878855 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878859 | controller | - ' logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878863 | controller | 2 == 2 '']''' 2026-03-09 19:10:03.878867 | controller | - ' logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878872 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878876 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878880 | controller | service' 2026-03-09 19:10:03.878885 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878889 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878893 | controller | - ' ]' 2026-03-09 19:10:03.878898 | controller | - ' logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878902 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878906 | controller | - ' logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878910 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878915 | controller | - ' logger.go:42: 19:07:34 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878919 | controller | 2 == 2 '']''' 2026-03-09 19:10:03.878923 | controller | - ' logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878927 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878938 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.878942 | controller | service' 2026-03-09 19:10:03.878946 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.878951 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.878955 | controller | - ' ]' 2026-03-09 19:10:03.878959 | controller | - ' logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.878964 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.878968 | controller | - ' logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.878972 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.878977 | controller | - ' logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.878983 | controller | 2 == 2 '']''' 2026-03-09 19:10:03.878987 | controller | - ' logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.878991 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.878995 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.879000 | controller | service' 2026-03-09 19:10:03.879004 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.879008 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.879013 | controller | - ' ]' 2026-03-09 19:10:03.879017 | controller | - ' logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.879021 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879025 | controller | - ' logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.879030 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879034 | controller | - ' logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.879038 | controller | 2 == 2 '']''' 2026-03-09 19:10:03.879043 | controller | - ' logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.879047 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879051 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.879055 | controller | service' 2026-03-09 19:10:03.879060 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.879064 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.879068 | controller | - ' ]' 2026-03-09 19:10:03.879073 | controller | - ' logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.879084 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879089 | controller | - ' logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.879093 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879098 | controller | - ' logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.879102 | controller | 2 == 2 '']''' 2026-03-09 19:10:03.879106 | controller | - ' logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.879112 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879117 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.879122 | controller | service' 2026-03-09 19:10:03.879126 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.879132 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.879137 | controller | - ' ]' 2026-03-09 19:10:03.879141 | controller | - ' logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.879157 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879163 | controller | - ' logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.879168 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879172 | controller | - ' logger.go:42: 19:07:39 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.879176 | controller | 2 == 2 '']''' 2026-03-09 19:10:03.879181 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-09 19:10:03.879185 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879189 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.879195 | controller | service' 2026-03-09 19:10:03.879200 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-09 19:10:03.879204 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.879208 | controller | - ' ]' 2026-03-09 19:10:03.879213 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-09 19:10:03.879217 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879221 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-09 19:10:03.879225 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879230 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-09 19:10:03.879234 | controller | 2 == 2 '']''' 2026-03-09 19:10:03.879238 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | test step 2026-03-09 19:10:03.879242 | controller | completed 1-deploy-watcher-no-cinder' 2026-03-09 19:10:03.879247 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | starting test step 2026-03-09 19:10:03.879251 | controller | 2-deploy-cinder' 2026-03-09 19:10:03.879255 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack 2026-03-09 19:10:03.879260 | controller | updated' 2026-03-09 19:10:03.879266 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879270 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879275 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879279 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879283 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879288 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879292 | controller | - ' ]' 2026-03-09 19:10:03.879296 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879301 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879307 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879311 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879316 | controller | - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879320 | controller | - ' logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879324 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879329 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879333 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879338 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879342 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879346 | controller | - ' ]' 2026-03-09 19:10:03.879353 | controller | - ' logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879358 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879362 | controller | - ' logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879367 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879371 | controller | - ' logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879384 | controller | - ' logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879388 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879395 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879399 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879403 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879408 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879412 | controller | - ' ]' 2026-03-09 19:10:03.879416 | controller | - ' logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879420 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879425 | controller | - ' logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879429 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879433 | controller | - ' logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879438 | controller | - ' logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879442 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879446 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879451 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879455 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879459 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879463 | controller | - ' ]' 2026-03-09 19:10:03.879468 | controller | - ' logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879472 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879478 | controller | - ' logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879482 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879487 | controller | - ' logger.go:42: 19:07:44 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879491 | controller | - ' logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879495 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879499 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879504 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879508 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879512 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879517 | controller | - ' ]' 2026-03-09 19:10:03.879521 | controller | - ' logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879525 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879529 | controller | - ' logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879534 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879538 | controller | - ' logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879542 | controller | - ' logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879546 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879551 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879580 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879589 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879594 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879600 | controller | - ' ]' 2026-03-09 19:10:03.879607 | controller | - ' logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879616 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879622 | controller | - ' logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879628 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879633 | controller | - ' logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879639 | controller | - ' logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879645 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879654 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879659 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879663 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879667 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879671 | controller | - ' ]' 2026-03-09 19:10:03.879675 | controller | - ' logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879680 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879684 | controller | - ' logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879697 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879702 | controller | - ' logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879707 | controller | - ' logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879713 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879719 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879725 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879731 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879737 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879743 | controller | - ' ]' 2026-03-09 19:10:03.879749 | controller | - ' logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879755 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879761 | controller | - ' logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879767 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879771 | controller | - ' logger.go:42: 19:07:49 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879776 | controller | - ' logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879780 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879784 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879788 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879793 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879797 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879802 | controller | - ' ]' 2026-03-09 19:10:03.879809 | controller | - ' logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879814 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879820 | controller | - ' logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879826 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879832 | controller | - ' logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879838 | controller | - ' logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879847 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879853 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879858 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879862 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879866 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879871 | controller | - ' ]' 2026-03-09 19:10:03.879875 | controller | - ' logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879879 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879883 | controller | - ' logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879888 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879893 | controller | - ' logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879899 | controller | - ' logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879905 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879911 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879917 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879923 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879929 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.879934 | controller | - ' ]' 2026-03-09 19:10:03.879940 | controller | - ' logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.879946 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.879952 | controller | - ' logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.879958 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.879964 | controller | - ' logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.879969 | controller | - ' logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.879973 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.879978 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.879982 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.879988 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.879994 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880000 | controller | - ' ]' 2026-03-09 19:10:03.880010 | controller | - ' logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880026 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880033 | controller | - ' logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880039 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880045 | controller | - ' logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880053 | controller | - ' logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880059 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880064 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880070 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880078 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880084 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880092 | controller | - ' ]' 2026-03-09 19:10:03.880098 | controller | - ' logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880103 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880107 | controller | - ' logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880111 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880115 | controller | - ' logger.go:42: 19:07:55 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880120 | controller | - ' logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880124 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880128 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880132 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880137 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880141 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880158 | controller | - ' ]' 2026-03-09 19:10:03.880164 | controller | - ' logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880169 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880173 | controller | - ' logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880177 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880181 | controller | - ' logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880186 | controller | - ' logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880190 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880194 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880198 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880203 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880207 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880211 | controller | - ' ]' 2026-03-09 19:10:03.880215 | controller | - ' logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880220 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880224 | controller | - ' logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880228 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880232 | controller | - ' logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880236 | controller | - ' logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880241 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880245 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880249 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880253 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880258 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880262 | controller | - ' ]' 2026-03-09 19:10:03.880266 | controller | - ' logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880270 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880283 | controller | - ' logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880289 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880295 | controller | - ' logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880299 | controller | - ' logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880304 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880308 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880312 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880316 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880321 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880325 | controller | - ' ]' 2026-03-09 19:10:03.880337 | controller | - ' logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880342 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880346 | controller | - ' logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880351 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880355 | controller | - ' logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880359 | controller | - ' logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880364 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880368 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880372 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880377 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880381 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880385 | controller | - ' ]' 2026-03-09 19:10:03.880390 | controller | - ' logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880394 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880398 | controller | - ' logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880402 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880407 | controller | - ' logger.go:42: 19:08:01 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880411 | controller | - ' logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880415 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880420 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880424 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880428 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880433 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880437 | controller | - ' ]' 2026-03-09 19:10:03.880441 | controller | - ' logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880446 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880452 | controller | - ' logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880456 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880461 | controller | - ' logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880465 | controller | - ' logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880469 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880474 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880478 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880484 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880488 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880493 | controller | - ' ]' 2026-03-09 19:10:03.880497 | controller | - ' logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880501 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880506 | controller | - ' logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880510 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880514 | controller | - ' logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880519 | controller | - ' logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880523 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880527 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880531 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880536 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880540 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880544 | controller | - ' ]' 2026-03-09 19:10:03.880549 | controller | - ' logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880553 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880557 | controller | - ' logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880562 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880566 | controller | - ' logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880570 | controller | - ' logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880574 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880579 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880583 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880594 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880599 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880604 | controller | - ' ]' 2026-03-09 19:10:03.880608 | controller | - ' logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880612 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880617 | controller | - ' logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880621 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880625 | controller | - ' logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880629 | controller | - ' logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880634 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880638 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880642 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880647 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880651 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880655 | controller | - ' ]' 2026-03-09 19:10:03.880660 | controller | - ' logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880664 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880668 | controller | - ' logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880675 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880679 | controller | - ' logger.go:42: 19:08:07 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880684 | controller | - ' logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880688 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880692 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880697 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880701 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880705 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880710 | controller | - ' ]' 2026-03-09 19:10:03.880714 | controller | - ' logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880718 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880722 | controller | - ' logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880727 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880731 | controller | - ' logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880736 | controller | - ' logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880740 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880746 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880753 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880759 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880765 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880771 | controller | - ' ]' 2026-03-09 19:10:03.880776 | controller | - ' logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880781 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880785 | controller | - ' logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880789 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880798 | controller | - ' logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880802 | controller | - ' logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880807 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880811 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880816 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880820 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880824 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880828 | controller | - ' ]' 2026-03-09 19:10:03.880833 | controller | - ' logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880837 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880843 | controller | - ' logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880848 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880852 | controller | - ' logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880856 | controller | - ' logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880861 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880868 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880874 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880879 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880883 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880887 | controller | - ' ]' 2026-03-09 19:10:03.880892 | controller | - ' logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880896 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880901 | controller | - ' logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880905 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880909 | controller | - ' logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880914 | controller | - ' logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880918 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880922 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880927 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880932 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880936 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880940 | controller | - ' ]' 2026-03-09 19:10:03.880945 | controller | - ' logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.880949 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.880955 | controller | - ' logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.880960 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.880964 | controller | - ' logger.go:42: 19:08:13 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.880968 | controller | - ' logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.880973 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.880977 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.880981 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.880985 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.880990 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.880994 | controller | - ' ]' 2026-03-09 19:10:03.880998 | controller | - ' logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881003 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881007 | controller | - ' logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881011 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881017 | controller | - ' logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881021 | controller | - ' logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881026 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881030 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881034 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881038 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881043 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881047 | controller | - ' ]' 2026-03-09 19:10:03.881051 | controller | - ' logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881057 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881062 | controller | - ' logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881066 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881070 | controller | - ' logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881074 | controller | - ' logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881079 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881083 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881087 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881092 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881096 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881100 | controller | - ' ]' 2026-03-09 19:10:03.881104 | controller | - ' logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881108 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881113 | controller | - ' logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881117 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881121 | controller | - ' logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881131 | controller | - ' logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881135 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881139 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881175 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881195 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881199 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881204 | controller | - ' ]' 2026-03-09 19:10:03.881208 | controller | - ' logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881212 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881217 | controller | - ' logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881221 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881225 | controller | - ' logger.go:42: 19:08:18 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881229 | controller | - ' logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881234 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881238 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881242 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881246 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881251 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881255 | controller | - ' ]' 2026-03-09 19:10:03.881259 | controller | - ' logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881263 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881268 | controller | - ' logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881272 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881276 | controller | - ' logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881280 | controller | - ' logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881287 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881291 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881295 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881300 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881304 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881308 | controller | - ' ]' 2026-03-09 19:10:03.881312 | controller | - ' logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881317 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881321 | controller | - ' logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881325 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881329 | controller | - ' logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881334 | controller | - ' logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881338 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881342 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881346 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881351 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881355 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881359 | controller | - ' ]' 2026-03-09 19:10:03.881363 | controller | - ' logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881367 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881372 | controller | - ' logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881376 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881380 | controller | - ' logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881384 | controller | - ' logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881389 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881393 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881397 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881401 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881406 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881410 | controller | - ' ]' 2026-03-09 19:10:03.881414 | controller | - ' logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881418 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881423 | controller | - ' logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881427 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881434 | controller | - ' logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881439 | controller | - ' logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881443 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881448 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881452 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881456 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881460 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881466 | controller | - ' ]' 2026-03-09 19:10:03.881471 | controller | - ' logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881475 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881479 | controller | - ' logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881483 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881488 | controller | - ' logger.go:42: 19:08:24 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881492 | controller | - ' logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881496 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881502 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881507 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881511 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881515 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881520 | controller | - ' ]' 2026-03-09 19:10:03.881524 | controller | - ' logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881528 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881532 | controller | - ' logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881537 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881541 | controller | - ' logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881545 | controller | - ' logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881549 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881554 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881558 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881562 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881566 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881571 | controller | - ' ]' 2026-03-09 19:10:03.881575 | controller | - ' logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881579 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881584 | controller | - ' logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881588 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881592 | controller | - ' logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881596 | controller | - ' logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881601 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881605 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881609 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881613 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881618 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881622 | controller | - ' ]' 2026-03-09 19:10:03.881626 | controller | - ' logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881630 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881636 | controller | - ' logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881640 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881646 | controller | - ' logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-09 19:10:03.881651 | controller | - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881655 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881659 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881663 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881667 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881672 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881676 | controller | - ' ]' 2026-03-09 19:10:03.881680 | controller | - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881684 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881691 | controller | - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881696 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881700 | controller | - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | Error from server 2026-03-09 19:10:03.881704 | controller | (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-09 19:10:03.881709 | controller | is waiting to start: ContainerCreating' 2026-03-09 19:10:03.881713 | controller | - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.881717 | controller | - ' logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881721 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881726 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881730 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881734 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881738 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881744 | controller | - ' ]' 2026-03-09 19:10:03.881750 | controller | - ' logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881756 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881762 | controller | - ' logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881768 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881773 | controller | - ' logger.go:42: 19:08:30 | watcher-cinder/2-deploy-cinder | Error from server 2026-03-09 19:10:03.881779 | controller | (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-09 19:10:03.881785 | controller | is waiting to start: ContainerCreating' 2026-03-09 19:10:03.881791 | controller | - ' logger.go:42: 19:08:30 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.881795 | controller | - ' logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881799 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881804 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881808 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881812 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881816 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881821 | controller | - ' ]' 2026-03-09 19:10:03.881825 | controller | - ' logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881829 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881833 | controller | - ' logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881840 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881844 | controller | - ' logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.881851 | controller | - ' logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881855 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881859 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881864 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881868 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881872 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881876 | controller | - ' ]' 2026-03-09 19:10:03.881881 | controller | - ' logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881885 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881889 | controller | - ' logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881893 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881898 | controller | - ' logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.881902 | controller | - ' logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881906 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881910 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881915 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881919 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881923 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881927 | controller | - ' ]' 2026-03-09 19:10:03.881932 | controller | - ' logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881936 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881940 | controller | - ' logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881944 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.881949 | controller | - ' logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.881956 | controller | - ' logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.881960 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.881965 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.881969 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.881973 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.881977 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.881982 | controller | - ' ]' 2026-03-09 19:10:03.881986 | controller | - ' logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.881990 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.881995 | controller | - ' logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.881999 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882003 | controller | - ' logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.882007 | controller | - ' logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.882012 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882016 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.882022 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.882026 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882030 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.882035 | controller | - ' ]' 2026-03-09 19:10:03.882039 | controller | - ' logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882043 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882047 | controller | - ' logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882052 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882056 | controller | - ' logger.go:42: 19:08:36 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.882060 | controller | - ' logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.882064 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882069 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.882073 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.882077 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882081 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.882086 | controller | - ' ]' 2026-03-09 19:10:03.882090 | controller | - ' logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882094 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882098 | controller | - ' logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882103 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882107 | controller | - ' logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.882111 | controller | - ' logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.882115 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882120 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.882124 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.882128 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882132 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.882137 | controller | - ' ]' 2026-03-09 19:10:03.882141 | controller | - ' logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882159 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882166 | controller | - ' logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882170 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882174 | controller | - ' logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.882179 | controller | - ' logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.882183 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882190 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.882195 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.882199 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882203 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.882208 | controller | - ' ]' 2026-03-09 19:10:03.882212 | controller | - ' logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882218 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882224 | controller | - ' logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882228 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882236 | controller | - ' logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.882240 | controller | - ' logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | running command: 2026-03-09 19:10:03.882245 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882249 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-09 19:10:03.882254 | controller | - ' # does not log that storage collector is skipped' 2026-03-09 19:10:03.882258 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882262 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-09 19:10:03.882266 | controller | - ' ]' 2026-03-09 19:10:03.882271 | controller | - ' logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882275 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882279 | controller | - ' logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882284 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882288 | controller | - ' logger.go:42: 19:08:41 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-09 19:10:03.882292 | controller | - ' logger.go:42: 19:08:41 | watcher-cinder/2-deploy-cinder | test step completed 2026-03-09 19:10:03.882297 | controller | 2-deploy-cinder' 2026-03-09 19:10:03.882301 | controller | - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | starting test step 2026-03-09 19:10:03.882305 | controller | 3-remove-cinder' 2026-03-09 19:10:03.882311 | controller | - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack 2026-03-09 19:10:03.882317 | controller | updated' 2026-03-09 19:10:03.882323 | controller | - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882329 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882335 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882341 | controller | service' 2026-03-09 19:10:03.882346 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882352 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882357 | controller | - ' ]' 2026-03-09 19:10:03.882364 | controller | - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882369 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882375 | controller | - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882381 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882386 | controller | - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882392 | controller | - ' logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882397 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882403 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882408 | controller | service' 2026-03-09 19:10:03.882413 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882421 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882426 | controller | - ' ]' 2026-03-09 19:10:03.882431 | controller | - ' logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882435 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882441 | controller | - ' logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882445 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882450 | controller | - ' logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882454 | controller | - ' logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882458 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882462 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882467 | controller | service' 2026-03-09 19:10:03.882471 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882475 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882479 | controller | - ' ]' 2026-03-09 19:10:03.882484 | controller | - ' logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882490 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882494 | controller | - ' logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882498 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882503 | controller | - ' logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882507 | controller | - ' logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882511 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882515 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882520 | controller | service' 2026-03-09 19:10:03.882524 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882528 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882536 | controller | - ' ]' 2026-03-09 19:10:03.882540 | controller | - ' logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882544 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882549 | controller | - ' logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882553 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882557 | controller | - ' logger.go:42: 19:08:45 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882563 | controller | - ' logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882568 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882572 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882576 | controller | service' 2026-03-09 19:10:03.882580 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882585 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882590 | controller | - ' ]' 2026-03-09 19:10:03.882595 | controller | - ' logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882599 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882603 | controller | - ' logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882607 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882612 | controller | - ' logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882616 | controller | - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882620 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882624 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882630 | controller | service' 2026-03-09 19:10:03.882635 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882639 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882643 | controller | - ' ]' 2026-03-09 19:10:03.882647 | controller | - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882652 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882656 | controller | - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882660 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882664 | controller | - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | Error from server 2026-03-09 19:10:03.882669 | controller | (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-09 19:10:03.882673 | controller | is waiting to start: ContainerCreating' 2026-03-09 19:10:03.882677 | controller | - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882681 | controller | - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882686 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882690 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882694 | controller | service' 2026-03-09 19:10:03.882698 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882703 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882707 | controller | - ' ]' 2026-03-09 19:10:03.882711 | controller | - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882715 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882720 | controller | - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882724 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882728 | controller | - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | Error from server 2026-03-09 19:10:03.882732 | controller | (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-09 19:10:03.882737 | controller | is waiting to start: ContainerCreating' 2026-03-09 19:10:03.882742 | controller | - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882748 | controller | - ' logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882754 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882760 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882766 | controller | service' 2026-03-09 19:10:03.882771 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882775 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882780 | controller | - ' ]' 2026-03-09 19:10:03.882784 | controller | - ' logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882788 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882792 | controller | - ' logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882796 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882801 | controller | - ' logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882808 | controller | - ' logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882812 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882816 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882820 | controller | service' 2026-03-09 19:10:03.882827 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882831 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882835 | controller | - ' ]' 2026-03-09 19:10:03.882840 | controller | - ' logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882844 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882848 | controller | - ' logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882852 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882856 | controller | - ' logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-09 19:10:03.882861 | controller | - ' logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882865 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882869 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882873 | controller | service' 2026-03-09 19:10:03.882877 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882882 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882886 | controller | - ' ]' 2026-03-09 19:10:03.882890 | controller | - ' logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882894 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882898 | controller | - ' logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882903 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882909 | controller | - ' logger.go:42: 19:08:53 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.882914 | controller | - ' logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882918 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882922 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882926 | controller | service' 2026-03-09 19:10:03.882931 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882935 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882939 | controller | - ' ]' 2026-03-09 19:10:03.882943 | controller | - ' logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882947 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.882952 | controller | - ' logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.882956 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.882960 | controller | - ' logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.882964 | controller | - ' logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.882968 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.882973 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.882977 | controller | service' 2026-03-09 19:10:03.882981 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.882985 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.882989 | controller | - ' ]' 2026-03-09 19:10:03.882994 | controller | - ' logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.882998 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.883002 | controller | - ' logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.883006 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.883012 | controller | - ' logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.883017 | controller | - ' logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.883021 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.883025 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.883029 | controller | service' 2026-03-09 19:10:03.883033 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.883038 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.883042 | controller | - ' ]' 2026-03-09 19:10:03.883046 | controller | - ' logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.883050 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.883054 | controller | - ' logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.883059 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.883063 | controller | - ' logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.883067 | controller | - ' logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.883071 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.883078 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.883082 | controller | service' 2026-03-09 19:10:03.883087 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.883091 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.883095 | controller | - ' ]' 2026-03-09 19:10:03.883099 | controller | - ' logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.883103 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.883108 | controller | - ' logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.883112 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.883116 | controller | - ' logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.883120 | controller | - ' logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.883124 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.883129 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.883133 | controller | service' 2026-03-09 19:10:03.883137 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.883141 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.883158 | controller | - ' ]' 2026-03-09 19:10:03.883165 | controller | - ' logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.883180 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.883184 | controller | - ' logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.883189 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.883193 | controller | - ' logger.go:42: 19:08:59 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.883197 | controller | - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | running command: 2026-03-09 19:10:03.883201 | controller | [sh -c set -euxo pipefail' 2026-03-09 19:10:03.883206 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-09 19:10:03.883210 | controller | service' 2026-03-09 19:10:03.883214 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-09 19:10:03.883220 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-09 19:10:03.883225 | controller | - ' ]' 2026-03-09 19:10:03.883229 | controller | - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-09 19:10:03.883233 | controller | watcher-kuttl-decision-engine-0' 2026-03-09 19:10:03.883237 | controller | - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-09 19:10:03.883242 | controller | storage service is not enabled, skipping storage collector''' 2026-03-09 19:10:03.883246 | controller | - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-09 19:10:03.883252 | controller | - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | test step completed 2026-03-09 19:10:03.883256 | controller | 3-remove-cinder' 2026-03-09 19:10:03.883260 | controller | - ' logger.go:42: 19:09:00 | watcher-cinder/4-cleanup-watcher | starting test step 2026-03-09 19:10:03.883265 | controller | 4-cleanup-watcher' 2026-03-09 19:10:03.883269 | controller | - ' logger.go:42: 19:09:08 | watcher-cinder/4-cleanup-watcher | test step completed 2026-03-09 19:10:03.883273 | controller | 4-cleanup-watcher' 2026-03-09 19:10:03.883277 | controller | - ' logger.go:42: 19:09:08 | watcher-cinder | skipping kubernetes event logging' 2026-03-09 19:10:03.883282 | controller | - === CONT kuttl/harness/watcher-api-scaling 2026-03-09 19:10:03.883286 | controller | - ' logger.go:42: 19:09:08 | watcher-api-scaling | Skipping creation of user-supplied 2026-03-09 19:10:03.883290 | controller | namespace: watcher-kuttl-default' 2026-03-09 19:10:03.883294 | controller | - ' logger.go:42: 19:09:08 | watcher-api-scaling/0-cleanup-watcher | starting test 2026-03-09 19:10:03.883299 | controller | step 0-cleanup-watcher' 2026-03-09 19:10:03.883303 | controller | - ' logger.go:42: 19:09:08 | watcher-api-scaling/0-cleanup-watcher | test step 2026-03-09 19:10:03.883307 | controller | completed 0-cleanup-watcher' 2026-03-09 19:10:03.883311 | controller | - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | starting 2026-03-09 19:10:03.883316 | controller | test step 1-deploy-with-defaults' 2026-03-09 19:10:03.883320 | controller | - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-09 19:10:03.883324 | controller | created' 2026-03-09 19:10:03.883328 | controller | - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-09 19:10:03.883332 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.883337 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.883341 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.883345 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.883349 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.883354 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.883358 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.883365 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.883369 | controller | ]' 2026-03-09 19:10:03.883374 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.883378 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.883382 | controller | == "" ]; then' 2026-03-09 19:10:03.883389 | controller | - ' exit 0' 2026-03-09 19:10:03.883393 | controller | - ' fi' 2026-03-09 19:10:03.883397 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.883402 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.883408 | controller | - ' counter=0' 2026-03-09 19:10:03.883415 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.883426 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.883432 | controller | - ' echo ${i}' 2026-03-09 19:10:03.883438 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.883444 | controller | - ' fi' 2026-03-09 19:10:03.883449 | controller | - ' done' 2026-03-09 19:10:03.883455 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.883460 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.883466 | controller | - ' exit 1' 2026-03-09 19:10:03.883472 | controller | - ' else' 2026-03-09 19:10:03.883478 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.883483 | controller | - ' fi' 2026-03-09 19:10:03.883487 | controller | - ' ]' 2026-03-09 19:10:03.883491 | controller | - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-09 19:10:03.883496 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.883500 | controller | -c Name -c Type' 2026-03-09 19:10:03.883504 | controller | - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.883508 | controller | -c ''^watcher''' 2026-03-09 19:10:03.883512 | controller | - ' logger.go:42: 19:09:11 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.883517 | controller | 0 == 1 '']''' 2026-03-09 19:10:03.883522 | controller | - ' logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-09 19:10:03.883528 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.883533 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.883539 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.883545 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.883568 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.883576 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.883582 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.883588 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.883594 | controller | ]' 2026-03-09 19:10:03.883600 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.883604 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.883608 | controller | == "" ]; then' 2026-03-09 19:10:03.883612 | controller | - ' exit 0' 2026-03-09 19:10:03.883617 | controller | - ' fi' 2026-03-09 19:10:03.883621 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.883628 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.883633 | controller | - ' counter=0' 2026-03-09 19:10:03.883637 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.883641 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.883645 | controller | - ' echo ${i}' 2026-03-09 19:10:03.883650 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.883654 | controller | - ' fi' 2026-03-09 19:10:03.883658 | controller | - ' done' 2026-03-09 19:10:03.883662 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.883667 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.883671 | controller | - ' exit 1' 2026-03-09 19:10:03.883675 | controller | - ' else' 2026-03-09 19:10:03.883679 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.883684 | controller | - ' fi' 2026-03-09 19:10:03.883688 | controller | - ' ]' 2026-03-09 19:10:03.883692 | controller | - ' logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-09 19:10:03.883699 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.883703 | controller | -c Name -c Type' 2026-03-09 19:10:03.883707 | controller | - ' logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.883711 | controller | -c ''^watcher''' 2026-03-09 19:10:03.883716 | controller | - ' logger.go:42: 19:09:14 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.883720 | controller | 0 == 1 '']''' 2026-03-09 19:10:03.883724 | controller | - ' logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-09 19:10:03.883728 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.883733 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.883737 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.883742 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.883748 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.883754 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.883765 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.883771 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.883777 | controller | ]' 2026-03-09 19:10:03.883783 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.883788 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.883793 | controller | == "" ]; then' 2026-03-09 19:10:03.883797 | controller | - ' exit 0' 2026-03-09 19:10:03.883801 | controller | - ' fi' 2026-03-09 19:10:03.883821 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.883826 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.883830 | controller | - ' counter=0' 2026-03-09 19:10:03.883834 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.883838 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.883843 | controller | - ' echo ${i}' 2026-03-09 19:10:03.883850 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.883854 | controller | - ' fi' 2026-03-09 19:10:03.883858 | controller | - ' done' 2026-03-09 19:10:03.883863 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.883875 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.883879 | controller | - ' exit 1' 2026-03-09 19:10:03.883884 | controller | - ' else' 2026-03-09 19:10:03.883888 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.883892 | controller | - ' fi' 2026-03-09 19:10:03.883897 | controller | - ' ]' 2026-03-09 19:10:03.883903 | controller | - ' logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-09 19:10:03.883907 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.883912 | controller | -c Name -c Type' 2026-03-09 19:10:03.883916 | controller | - ' logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.883920 | controller | -c ''^watcher''' 2026-03-09 19:10:03.883924 | controller | - ' logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.883929 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.883933 | controller | - ' logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.883937 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.883941 | controller | -c Name -c Type -c ID' 2026-03-09 19:10:03.883946 | controller | - ' logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-09 19:10:03.883952 | controller | ''{print $1}''' 2026-03-09 19:10:03.883956 | controller | - ' logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.883961 | controller | watcher' 2026-03-09 19:10:03.883965 | controller | - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d' 2026-03-09 19:10:03.883969 | controller | - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.883974 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.883978 | controller | - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.883982 | controller | 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d '']''' 2026-03-09 19:10:03.883987 | controller | - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.883991 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.883995 | controller | - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.883999 | controller | -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.884004 | controller | '']''' 2026-03-09 19:10:03.884008 | controller | - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884012 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.884016 | controller | - ' logger.go:42: 19:09:21 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884021 | controller | '''' == '''' '']''' 2026-03-09 19:10:03.884025 | controller | - ' logger.go:42: 19:09:21 | watcher-api-scaling/1-deploy-with-defaults | + exit 2026-03-09 19:10:03.884029 | controller | 0' 2026-03-09 19:10:03.884033 | controller | - ' logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-09 19:10:03.884038 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.884042 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.884046 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.884051 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.884055 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.884059 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.884063 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.884068 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.884072 | controller | ]' 2026-03-09 19:10:03.884076 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.884083 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.884087 | controller | == "" ]; then' 2026-03-09 19:10:03.884092 | controller | - ' exit 0' 2026-03-09 19:10:03.884096 | controller | - ' fi' 2026-03-09 19:10:03.884100 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.884104 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.884109 | controller | - ' counter=0' 2026-03-09 19:10:03.884113 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.884117 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.884122 | controller | - ' echo ${i}' 2026-03-09 19:10:03.884126 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.884130 | controller | - ' fi' 2026-03-09 19:10:03.884134 | controller | - ' done' 2026-03-09 19:10:03.884140 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.884158 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.884166 | controller | - ' exit 1' 2026-03-09 19:10:03.884170 | controller | - ' else' 2026-03-09 19:10:03.884174 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.884178 | controller | - ' fi' 2026-03-09 19:10:03.884183 | controller | - ' ]' 2026-03-09 19:10:03.884187 | controller | - ' logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-09 19:10:03.884191 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.884195 | controller | -c Name -c Type' 2026-03-09 19:10:03.884200 | controller | - ' logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.884204 | controller | -c ''^watcher''' 2026-03-09 19:10:03.884208 | controller | - ' logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884212 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.884216 | controller | - ' logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.884221 | controller | watcher' 2026-03-09 19:10:03.884225 | controller | - ' logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-09 19:10:03.884229 | controller | ''{print $1}''' 2026-03-09 19:10:03.884233 | controller | - ' logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884238 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.884244 | controller | -c Name -c Type -c ID' 2026-03-09 19:10:03.884250 | controller | - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d' 2026-03-09 19:10:03.884256 | controller | - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884262 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.884270 | controller | - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884276 | controller | 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d '']''' 2026-03-09 19:10:03.884282 | controller | - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884287 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.884291 | controller | - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884296 | controller | -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.884300 | controller | '']''' 2026-03-09 19:10:03.884304 | controller | - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884308 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.884313 | controller | - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884317 | controller | '''' == '''' '']''' 2026-03-09 19:10:03.884321 | controller | - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + exit 2026-03-09 19:10:03.884325 | controller | 0' 2026-03-09 19:10:03.884330 | controller | - ' logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-09 19:10:03.884334 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.884338 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.884342 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.884347 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.884351 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.884357 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.884361 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.884366 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.884370 | controller | ]' 2026-03-09 19:10:03.884374 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.884378 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.884383 | controller | == "" ]; then' 2026-03-09 19:10:03.884387 | controller | - ' exit 0' 2026-03-09 19:10:03.884393 | controller | - ' fi' 2026-03-09 19:10:03.884398 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.884402 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.884408 | controller | - ' counter=0' 2026-03-09 19:10:03.884416 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.884420 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.884425 | controller | - ' echo ${i}' 2026-03-09 19:10:03.884429 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.884433 | controller | - ' fi' 2026-03-09 19:10:03.884437 | controller | - ' done' 2026-03-09 19:10:03.884442 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.884446 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.884450 | controller | - ' exit 1' 2026-03-09 19:10:03.884454 | controller | - ' else' 2026-03-09 19:10:03.884459 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.884463 | controller | - ' fi' 2026-03-09 19:10:03.884467 | controller | - ' ]' 2026-03-09 19:10:03.884471 | controller | - ' logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-09 19:10:03.884476 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.884480 | controller | -c Name -c Type' 2026-03-09 19:10:03.884484 | controller | - ' logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.884488 | controller | -c ''^watcher''' 2026-03-09 19:10:03.884493 | controller | - ' logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884497 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.884501 | controller | - ' logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884505 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.884510 | controller | -c Name -c Type -c ID' 2026-03-09 19:10:03.884514 | controller | - ' logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-09 19:10:03.884518 | controller | ''{print $1}''' 2026-03-09 19:10:03.884522 | controller | - ' logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.884527 | controller | watcher' 2026-03-09 19:10:03.884531 | controller | - ' logger.go:42: 19:09:31 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d' 2026-03-09 19:10:03.884535 | controller | - ' logger.go:42: 19:09:31 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884539 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.884544 | controller | - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884548 | controller | 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d '']''' 2026-03-09 19:10:03.884552 | controller | - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884556 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.884562 | controller | - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884567 | controller | -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.884571 | controller | '']''' 2026-03-09 19:10:03.884577 | controller | - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884581 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.884585 | controller | - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884589 | controller | '''' == '''' '']''' 2026-03-09 19:10:03.884594 | controller | - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + exit 2026-03-09 19:10:03.884598 | controller | 0' 2026-03-09 19:10:03.884602 | controller | - ' logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-09 19:10:03.884607 | controller | command: [sh -c set -euxo pipefail' 2026-03-09 19:10:03.884613 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.884617 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-09 19:10:03.884621 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-09 19:10:03.884626 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-09 19:10:03.884630 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-09 19:10:03.884634 | controller | == ${SERVICEID} ]' 2026-03-09 19:10:03.884638 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-09 19:10:03.884643 | controller | ]' 2026-03-09 19:10:03.884647 | controller | - ' # If we are running the container locally, skip following test' 2026-03-09 19:10:03.884651 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-09 19:10:03.884655 | controller | == "" ]; then' 2026-03-09 19:10:03.884659 | controller | - ' exit 0' 2026-03-09 19:10:03.884664 | controller | - ' fi' 2026-03-09 19:10:03.884668 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-09 19:10:03.884672 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-09 19:10:03.884676 | controller | - ' counter=0' 2026-03-09 19:10:03.884681 | controller | - ' for i in ${env_variables}; do' 2026-03-09 19:10:03.884685 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-09 19:10:03.884689 | controller | - ' echo ${i}' 2026-03-09 19:10:03.884694 | controller | - ' counter=$((counter + 1))' 2026-03-09 19:10:03.884698 | controller | - ' fi' 2026-03-09 19:10:03.884702 | controller | - ' done' 2026-03-09 19:10:03.884706 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-09 19:10:03.884713 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-09 19:10:03.884717 | controller | - ' exit 1' 2026-03-09 19:10:03.884721 | controller | - ' else' 2026-03-09 19:10:03.884726 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-09 19:10:03.884730 | controller | - ' fi' 2026-03-09 19:10:03.884734 | controller | - ' ]' 2026-03-09 19:10:03.884739 | controller | - ' logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-09 19:10:03.884744 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.884750 | controller | -c Name -c Type' 2026-03-09 19:10:03.884756 | controller | - ' logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.884763 | controller | -c ''^watcher''' 2026-03-09 19:10:03.884768 | controller | - ' logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884774 | controller | 1 == 1 '']''' 2026-03-09 19:10:03.884780 | controller | - ' logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-09 19:10:03.884786 | controller | watcher' 2026-03-09 19:10:03.884791 | controller | - ' logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-09 19:10:03.884795 | controller | ''{print $1}''' 2026-03-09 19:10:03.884799 | controller | - ' logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884804 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-09 19:10:03.884808 | controller | -c Name -c Type -c ID' 2026-03-09 19:10:03.884812 | controller | - ' logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d' 2026-03-09 19:10:03.884816 | controller | - ' logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884821 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-09 19:10:03.884825 | controller | - ' logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884829 | controller | 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d '']''' 2026-03-09 19:10:03.884833 | controller | - ' logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884838 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-09 19:10:03.884842 | controller | - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884846 | controller | -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q 2026-03-09 19:10:03.884850 | controller | '']''' 2026-03-09 19:10:03.884854 | controller | - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-09 19:10:03.884859 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-09 19:10:03.884863 | controller | - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-09 19:10:03.884867 | controller | '''' == '''' '']''' 2026-03-09 19:10:03.884871 | controller | - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + exit 2026-03-09 19:10:03.884876 | controller | 0' 2026-03-09 19:10:03.884880 | controller | - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | test 2026-03-09 19:10:03.884884 | controller | step completed 1-deploy-with-defaults' 2026-03-09 19:10:03.884888 | controller | - ' logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | starting 2026-03-09 19:10:03.884893 | controller | test step 2-scale-up-watcher-api' 2026-03-09 19:10:03.884897 | controller | - ' logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | running 2026-03-09 19:10:03.884901 | controller | command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": 2026-03-09 19:10:03.884905 | controller | "replace", "path": "/spec/apiServiceTemplate/replicas", "value":3}]''' 2026-03-09 19:10:03.884910 | controller | - ' ]' 2026-03-09 19:10:03.884917 | controller | - ' logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.884921 | controller | patched' 2026-03-09 19:10:03.884926 | controller | - ' logger.go:42: 19:09:49 | watcher-api-scaling/2-scale-up-watcher-api | test 2026-03-09 19:10:03.884930 | controller | step completed 2-scale-up-watcher-api' 2026-03-09 19:10:03.884934 | controller | - ' logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | starting 2026-03-09 19:10:03.884938 | controller | test step 3-scale-down-watcher-api' 2026-03-09 19:10:03.884943 | controller | - ' logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | running 2026-03-09 19:10:03.884947 | controller | command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": 2026-03-09 19:10:03.884951 | controller | "replace", "path": "/spec/apiServiceTemplate/replicas", "value":1}]''' 2026-03-09 19:10:03.884955 | controller | - ' ]' 2026-03-09 19:10:03.884963 | controller | - ' logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | watcher.watcher.openstack.org/watcher-kuttl 2026-03-09 19:10:03.884967 | controller | patched' 2026-03-09 19:10:03.884972 | controller | - ' logger.go:42: 19:09:52 | watcher-api-scaling/3-scale-down-watcher-api | test 2026-03-09 19:10:03.884976 | controller | step completed 3-scale-down-watcher-api' 2026-03-09 19:10:03.884980 | controller | - ' logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-09 19:10:03.884984 | controller | | starting test step 4-scale-down-zero-watcher-api' 2026-03-09 19:10:03.884989 | controller | - ' logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-09 19:10:03.884996 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' 2026-03-09 19:10:03.885002 | controller | -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/replicas", "value":0}]''' 2026-03-09 19:10:03.885006 | controller | - ' ]' 2026-03-09 19:10:03.885011 | controller | - ' logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-09 19:10:03.885015 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched' 2026-03-09 19:10:03.885019 | controller | - ' logger.go:42: 19:09:53 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-09 19:10:03.885024 | controller | | test step completed 4-scale-down-zero-watcher-api' 2026-03-09 19:10:03.885028 | controller | - ' logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | starting test 2026-03-09 19:10:03.885032 | controller | step 5-cleanup-watcher' 2026-03-09 19:10:03.885036 | controller | - ' logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | running command: 2026-03-09 19:10:03.885041 | controller | [sh -c set -ex' 2026-03-09 19:10:03.885045 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.885049 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.885053 | controller | - ' ]' 2026-03-09 19:10:03.885058 | controller | - ' logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | + oc exec 2026-03-09 19:10:03.885062 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.885066 | controller | -c Type' 2026-03-09 19:10:03.885070 | controller | - ' logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c 2026-03-09 19:10:03.885075 | controller | ''^watcher''' 2026-03-09 19:10:03.885079 | controller | - ' logger.go:42: 19:09:56 | watcher-api-scaling/5-cleanup-watcher | + ''['' 1 2026-03-09 19:10:03.885083 | controller | == 0 '']''' 2026-03-09 19:10:03.885087 | controller | - ' logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | running command: 2026-03-09 19:10:03.885092 | controller | [sh -c set -ex' 2026-03-09 19:10:03.885096 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.885100 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.885104 | controller | - ' ]' 2026-03-09 19:10:03.885108 | controller | - ' logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | + oc exec 2026-03-09 19:10:03.885128 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.885133 | controller | -c Type' 2026-03-09 19:10:03.885138 | controller | - ' logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c 2026-03-09 19:10:03.885142 | controller | ''^watcher''' 2026-03-09 19:10:03.885160 | controller | - ' logger.go:42: 19:09:59 | watcher-api-scaling/5-cleanup-watcher | + ''['' 0 2026-03-09 19:10:03.885165 | controller | == 0 '']''' 2026-03-09 19:10:03.885180 | controller | - ' logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | running command: 2026-03-09 19:10:03.885184 | controller | [sh -c set -ex' 2026-03-09 19:10:03.885189 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-09 19:10:03.885193 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-09 19:10:03.885197 | controller | - ' ]' 2026-03-09 19:10:03.885201 | controller | - ' logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | + oc exec 2026-03-09 19:10:03.885208 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-09 19:10:03.885212 | controller | -c Type' 2026-03-09 19:10:03.885217 | controller | - ' logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c 2026-03-09 19:10:03.885221 | controller | ''^watcher''' 2026-03-09 19:10:03.885225 | controller | - ' logger.go:42: 19:10:02 | watcher-api-scaling/5-cleanup-watcher | + ''['' 0 2026-03-09 19:10:03.885229 | controller | == 0 '']''' 2026-03-09 19:10:03.885234 | controller | - ' logger.go:42: 19:10:02 | watcher-api-scaling/5-cleanup-watcher | test step 2026-03-09 19:10:03.885238 | controller | completed 5-cleanup-watcher' 2026-03-09 19:10:03.885242 | controller | - ' logger.go:42: 19:10:02 | watcher-api-scaling | skipping kubernetes event logging' 2026-03-09 19:10:03.885246 | controller | - === CONT kuttl/harness/deps 2026-03-09 19:10:03.885251 | controller | - ' logger.go:42: 19:10:02 | deps | Ignoring infra.yaml as it does not match file 2026-03-09 19:10:03.885255 | controller | name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.885259 | controller | - ' logger.go:42: 19:10:02 | deps | Ignoring keystone.yaml as it does not match 2026-03-09 19:10:03.885263 | controller | file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.885267 | controller | - ' logger.go:42: 19:10:02 | deps | Ignoring kustomization.yaml as it does not 2026-03-09 19:10:03.885272 | controller | match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.885276 | controller | - ' logger.go:42: 19:10:02 | deps | Ignoring namespace.yaml as it does not match 2026-03-09 19:10:03.885282 | controller | file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.885299 | controller | - ' logger.go:42: 19:10:02 | deps | Ignoring telemetry.yaml as it does not match 2026-03-09 19:10:03.885303 | controller | file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.885307 | controller | - ' logger.go:42: 19:10:02 | deps | Skipping creation of user-supplied namespace: 2026-03-09 19:10:03.885311 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.885316 | controller | - ' logger.go:42: 19:10:02 | deps | skipping kubernetes event logging' 2026-03-09 19:10:03.885320 | controller | - === CONT kuttl/harness/common 2026-03-09 19:10:03.885325 | controller | - ' logger.go:42: 19:10:02 | common | Ignoring cleanup-assert.yaml as it does not 2026-03-09 19:10:03.885331 | controller | match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.885339 | controller | - ' logger.go:42: 19:10:02 | common | Ignoring cleanup-errors.yaml as it does not 2026-03-09 19:10:03.885349 | controller | match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.961202 | controller | - ' logger.go:42: 19:10:02 | common | Ignoring cleanup-watcher.yaml as it does 2026-03-09 19:10:03.961245 | controller | not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.961253 | controller | - ' logger.go:42: 19:10:02 | common | Ignoring deploy-with-defaults.yaml as it 2026-03-09 19:10:03.961260 | controller | does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-09 19:10:03.961267 | controller | - ' logger.go:42: 19:10:02 | common | Skipping creation of user-supplied namespace: 2026-03-09 19:10:03.961287 | controller | watcher-kuttl-default' 2026-03-09 19:10:03.961295 | controller | - ' logger.go:42: 19:10:02 | common | skipping kubernetes event logging' 2026-03-09 19:10:03.961302 | controller | - === NAME kuttl 2026-03-09 19:10:03.961309 | controller | - ' harness.go:406: run tests finished' 2026-03-09 19:10:03.961315 | controller | - ' harness.go:514: cleaning up' 2026-03-09 19:10:03.961322 | controller | - ' harness.go:571: removing temp folder: ""' 2026-03-09 19:10:03.961328 | controller | - '--- FAIL: kuttl (1502.65s)' 2026-03-09 19:10:03.961335 | controller | - ' --- FAIL: kuttl/harness (0.00s)' 2026-03-09 19:10:03.961341 | controller | - ' --- FAIL: kuttl/harness/appcred-tests (959.40s)' 2026-03-09 19:10:03.961348 | controller | - ' --- PASS: kuttl/harness/watcher-notification (36.28s)' 2026-03-09 19:10:03.961354 | controller | - ' --- PASS: kuttl/harness/watcher-topology (33.59s)' 2026-03-09 19:10:03.961368 | controller | - ' --- PASS: kuttl/harness/watcher-tls-certs-change (37.89s)' 2026-03-09 19:10:03.961375 | controller | - ' --- PASS: kuttl/harness/watcher-tls (120.05s)' 2026-03-09 19:10:03.961381 | controller | - ' --- PASS: kuttl/harness/watcher-rmquser (34.79s)' 2026-03-09 19:10:03.961387 | controller | - ' --- PASS: kuttl/harness/watcher (114.14s)' 2026-03-09 19:10:03.961394 | controller | - ' --- PASS: kuttl/harness/watcher-cinder (112.06s)' 2026-03-09 19:10:03.961400 | controller | - ' --- PASS: kuttl/harness/watcher-api-scaling (54.41s)' 2026-03-09 19:10:03.961407 | controller | - ' --- PASS: kuttl/harness/deps (0.00s)' 2026-03-09 19:10:03.961422 | controller | - ' --- PASS: kuttl/harness/common (0.00s)' 2026-03-09 19:10:03.961428 | controller | - FAIL 2026-03-09 19:10:03.961441 | controller | 2026-03-09 19:10:04.027741 | controller | NO MORE HOSTS LEFT ************************************************************* 2026-03-09 19:10:04.027860 | controller | 2026-03-09 19:10:04.027869 | controller | PLAY RECAP ********************************************************************* 2026-03-09 19:10:04.027874 | controller | controller : ok=2 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 2026-03-09 19:10:04.027879 | controller | 2026-03-09 19:10:04.027883 | controller | Monday 09 March 2026 19:10:03 +0000 (0:25:04.305) 0:25:04.762 ********** 2026-03-09 19:10:04.027888 | controller | =============================================================================== 2026-03-09 19:10:04.027892 | controller | run kuttl test suite from operator Makefile -------------------------- 1504.31s 2026-03-09 19:10:04.027897 | controller | run_hook : Loop on hooks for pre_kuttl_from_operator -------------------- 0.16s 2026-03-09 19:10:04.027901 | controller | run_hook : Assert single hooks are all mappings ------------------------- 0.15s 2026-03-09 19:10:04.027906 | controller | run_hook : Assert parameters are valid ---------------------------------- 0.07s 2026-03-09 19:10:04.027910 | controller | Run hooks before running kuttl tests ------------------------------------ 0.03s 2026-03-09 19:10:04.027914 | controller | Monday 09 March 2026 19:10:03 +0000 (0:25:04.305) 0:25:04.762 ********** 2026-03-09 19:10:04.027919 | controller | =============================================================================== 2026-03-09 19:10:04.027923 | controller | ansible.builtin.command ---------------------------------------------- 1504.31s 2026-03-09 19:10:04.027927 | controller | run_hook ---------------------------------------------------------------- 0.38s 2026-03-09 19:10:04.027932 | controller | ansible.builtin.include_role -------------------------------------------- 0.03s 2026-03-09 19:10:04.027936 | controller | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2026-03-09 19:10:04.027940 | controller | total ---------------------------------------------------------------- 1504.72s 2026-03-09 19:10:05.523270 | controller | ERROR 2026-03-09 19:10:05.523611 | controller | { 2026-03-09 19:10:05.523678 | controller | "delta": "0:25:05.368908", 2026-03-09 19:10:05.523719 | controller | "end": "2026-03-09 19:10:04.033864", 2026-03-09 19:10:05.523757 | controller | "msg": "non-zero return code", 2026-03-09 19:10:05.523793 | controller | "rc": 2, 2026-03-09 19:10:05.523828 | controller | "start": "2026-03-09 18:44:58.664956" 2026-03-09 19:10:05.523862 | controller | } failure 2026-03-09 19:10:06.180320 | 2026-03-09 19:10:06.180444 | PLAY RECAP 2026-03-09 19:10:06.180490 | controller | ok: 0 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 0 2026-03-09 19:10:06.180512 | 2026-03-09 19:10:06.373632 | RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-run.yaml@main] 2026-03-09 19:10:06.382550 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-collect-logs.yml@main] 2026-03-09 19:10:07.102505 | 2026-03-09 19:10:07.102616 | PLAY [Run ci/playbooks/e2e-collect-logs.yml] 2026-03-09 19:10:07.147208 | 2026-03-09 19:10:07.147319 | TASK [Filter out host if needed] 2026-03-09 19:10:07.158958 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller 2026-03-09 19:10:07.165476 | 2026-03-09 19:10:07.165540 | TASK [Filter out host if needed] 2026-03-09 19:10:07.181267 | 2026-03-09 19:10:07.181346 | TASK [Ensure we have the ci-framework on host] 2026-03-09 19:10:07.720828 | controller | ok 2026-03-09 19:10:07.727292 | 2026-03-09 19:10:07.727377 | TASK [End host if no basedir] 2026-03-09 19:10:07.735936 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller 2026-03-09 19:10:07.741913 | 2026-03-09 19:10:07.742046 | TASK [Read base centos-9 scenarios] 2026-03-09 19:10:07.771724 | controller | ok 2026-03-09 19:10:07.882280 | 2026-03-09 19:10:07.882401 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-09 19:10:07.906626 | controller | skipping: Conditional result was False 2026-03-09 19:10:07.941272 | 2026-03-09 19:10:07.941385 | TASK [cifmw_helpers : Check if file is available] 2026-03-09 19:10:08.201819 | controller | ok 2026-03-09 19:10:08.215294 | 2026-03-09 19:10:08.215448 | TASK [cifmw_helpers : Create temporary directory] 2026-03-09 19:10:08.252544 | controller | skipping: Conditional result was False 2026-03-09 19:10:08.268568 | 2026-03-09 19:10:08.268735 | TASK [cifmw_helpers : Read vars] 2026-03-09 19:10:08.310882 | 2026-03-09 19:10:08.311173 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-09 19:10:08.366292 | 2026-03-09 19:10:08.366560 | TASK [cifmw_helpers : Include vars] 2026-03-09 19:10:08.418213 | 2026-03-09 19:10:08.418494 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-09 19:10:08.445296 | controller | skipping: Conditional result was False 2026-03-09 19:10:08.472636 | 2026-03-09 19:10:08.472810 | TASK [run_hook : Assert parameters are valid] 2026-03-09 19:10:08.552035 | controller | ok: All assertions passed 2026-03-09 19:10:08.564792 | 2026-03-09 19:10:08.564926 | TASK [run_hook : Assert single hooks are all mappings] 2026-03-09 19:10:08.646785 | controller | ok: All assertions passed 2026-03-09 19:10:08.661114 | 2026-03-09 19:10:08.661267 | LOOP [run_hook : Loop on hooks for pre_logs] 2026-03-09 19:10:08.755974 | 2026-03-09 19:10:08.756278 | TASK [cifmw_setup : Ensure cifmw_basedir param is set] 2026-03-09 19:10:08.783121 | controller | skipping: Conditional result was False 2026-03-09 19:10:08.797550 | 2026-03-09 19:10:08.797693 | TASK [Read artifacts parameters dir and set as facts] 2026-03-09 19:10:08.829010 | controller | ok 2026-03-09 19:10:08.860644 | 2026-03-09 19:10:08.860807 | TASK [cifmw_helpers : Check directory is available] 2026-03-09 19:10:09.104759 | controller | ok 2026-03-09 19:10:09.119725 | 2026-03-09 19:10:09.119923 | TASK [cifmw_helpers : Find yaml files] 2026-03-09 19:10:09.519604 | controller | ok: All paths examined 2026-03-09 19:10:09.535483 | 2026-03-09 19:10:09.535628 | TASK [cifmw_helpers : Print available yaml files] 2026-03-09 19:10:09.596130 | controller | ok: Found yaml files to parse: ['/home/zuul/ci-framework-data/artifacts/parameters/zuul-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/install-yamls-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/custom-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/openshift-login-params.yml'] 2026-03-09 19:10:09.609410 | 2026-03-09 19:10:09.609549 | LOOP [cifmw_helpers : Create files on localhost and use include_vars] 2026-03-09 19:10:09.649069 | controller | Output suppressed because no_log was given 2026-03-09 19:10:09.649332 | 2026-03-09 19:10:09.662320 | controller | Output suppressed because no_log was given 2026-03-09 19:10:09.672247 | controller | Output suppressed because no_log was given 2026-03-09 19:10:09.680171 | controller | Output suppressed because no_log was given 2026-03-09 19:10:09.701582 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml 2026-03-09 19:10:09.708333 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml 2026-03-09 19:10:09.715002 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml 2026-03-09 19:10:09.721552 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml 2026-03-09 19:10:09.731293 | 2026-03-09 19:10:09.731355 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-09 19:10:09.765562 | controller | skipping: Conditional result was False 2026-03-09 19:10:09.775508 | 2026-03-09 19:10:09.775590 | TASK [cifmw_helpers : Check if file is available] 2026-03-09 19:10:10.009349 | controller | ok 2026-03-09 19:10:10.016229 | 2026-03-09 19:10:10.016296 | TASK [cifmw_helpers : Create temporary directory] 2026-03-09 19:10:10.438712 | controller -> localhost | changed 2026-03-09 19:10:10.456198 | 2026-03-09 19:10:10.456342 | TASK [cifmw_helpers : Read vars] 2026-03-09 19:10:10.813189 | controller | Output suppressed because no_log was given 2026-03-09 19:10:10.826080 | 2026-03-09 19:10:10.826221 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-09 19:10:11.564996 | controller -> localhost | Output suppressed because no_log was given 2026-03-09 19:10:11.571789 | 2026-03-09 19:10:11.571851 | TASK [cifmw_helpers : Include vars] 2026-03-09 19:10:11.611149 | controller -> localhost | Output suppressed because no_log was given 2026-03-09 19:10:11.625065 | 2026-03-09 19:10:11.625201 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-09 19:10:12.048288 | controller -> localhost | changed 2026-03-09 19:10:12.055770 | 2026-03-09 19:10:12.055849 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-09 19:10:12.080152 | controller | skipping: Conditional result was False 2026-03-09 19:10:12.087096 | 2026-03-09 19:10:12.087158 | TASK [cifmw_helpers : Check if file is available] 2026-03-09 19:10:12.339409 | controller | ok 2026-03-09 19:10:12.353350 | 2026-03-09 19:10:12.353500 | TASK [cifmw_helpers : Create temporary directory] 2026-03-09 19:10:12.645621 | controller -> localhost | changed 2026-03-09 19:10:12.660634 | 2026-03-09 19:10:12.660772 | TASK [cifmw_helpers : Read vars] 2026-03-09 19:10:12.937283 | controller | Output suppressed because no_log was given 2026-03-09 19:10:12.951212 | 2026-03-09 19:10:12.951349 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-09 19:10:13.460771 | controller -> localhost | Output suppressed because no_log was given 2026-03-09 19:10:13.475417 | 2026-03-09 19:10:13.475553 | TASK [cifmw_helpers : Include vars] 2026-03-09 19:10:13.517343 | controller -> localhost | Output suppressed because no_log was given 2026-03-09 19:10:13.531974 | 2026-03-09 19:10:13.532165 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-09 19:10:13.805200 | controller -> localhost | changed 2026-03-09 19:10:13.823232 | 2026-03-09 19:10:13.823377 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-09 19:10:13.851841 | controller | skipping: Conditional result was False 2026-03-09 19:10:13.914646 | 2026-03-09 19:10:13.914787 | TASK [cifmw_helpers : Check if file is available] 2026-03-09 19:10:14.148908 | controller | ok 2026-03-09 19:10:14.164458 | 2026-03-09 19:10:14.164631 | TASK [cifmw_helpers : Create temporary directory] 2026-03-09 19:10:14.459743 | controller -> localhost | changed 2026-03-09 19:10:14.477767 | 2026-03-09 19:10:14.477933 | TASK [cifmw_helpers : Read vars] 2026-03-09 19:10:14.707972 | controller | Output suppressed because no_log was given 2026-03-09 19:10:14.721004 | 2026-03-09 19:10:14.721174 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-09 19:10:15.250442 | controller -> localhost | Output suppressed because no_log was given 2026-03-09 19:10:15.265684 | 2026-03-09 19:10:15.265880 | TASK [cifmw_helpers : Include vars] 2026-03-09 19:10:15.304656 | controller -> localhost | Output suppressed because no_log was given 2026-03-09 19:10:15.320082 | 2026-03-09 19:10:15.320294 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-09 19:10:15.595501 | controller -> localhost | changed 2026-03-09 19:10:15.621232 | 2026-03-09 19:10:15.621423 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-09 19:10:15.659661 | controller | skipping: Conditional result was False 2026-03-09 19:10:15.669600 | 2026-03-09 19:10:15.669856 | TASK [cifmw_helpers : Check if file is available] 2026-03-09 19:10:15.919379 | controller | ok 2026-03-09 19:10:15.928942 | 2026-03-09 19:10:15.929053 | TASK [cifmw_helpers : Create temporary directory] 2026-03-09 19:10:16.211054 | controller -> localhost | changed 2026-03-09 19:10:16.220894 | 2026-03-09 19:10:16.221002 | TASK [cifmw_helpers : Read vars] 2026-03-09 19:10:16.477953 | controller | Output suppressed because no_log was given 2026-03-09 19:10:16.491627 | 2026-03-09 19:10:16.492127 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-09 19:10:16.921743 | controller -> localhost | Output suppressed because no_log was given 2026-03-09 19:10:16.937993 | 2026-03-09 19:10:16.938229 | TASK [cifmw_helpers : Include vars] 2026-03-09 19:10:16.997556 | controller -> localhost | Output suppressed because no_log was given 2026-03-09 19:10:17.014369 | 2026-03-09 19:10:17.014581 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-09 19:10:17.246779 | controller -> localhost | changed 2026-03-09 19:10:17.276899 | 2026-03-09 19:10:17.277119 | TASK [cifmw_setup : Set custom cifmw PATH reusable fact] 2026-03-09 19:10:17.305854 | controller | skipping: Conditional result was False 2026-03-09 19:10:17.323071 | 2026-03-09 19:10:17.323302 | TASK [cifmw_setup : Set destination folder for the logs] 2026-03-09 19:10:17.376164 | controller | ok 2026-03-09 19:10:17.390436 | 2026-03-09 19:10:17.390633 | TASK [artifacts : Verify if "success" flag exists after successful tests execution] 2026-03-09 19:10:17.586362 | controller | ok 2026-03-09 19:10:17.604790 | 2026-03-09 19:10:17.605007 | TASK [artifacts : Ensure we have at least minimal ansible facts] 2026-03-09 19:10:18.195672 | controller | ok 2026-03-09 19:10:18.205954 | 2026-03-09 19:10:18.206081 | LOOP [artifacts : Ensure base directory exists] 2026-03-09 19:10:18.476972 | controller | ok: "artifacts" 2026-03-09 19:10:18.660563 | controller | ok: "logs" 2026-03-09 19:10:18.678181 | 2026-03-09 19:10:18.678338 | TASK [artifacts : Load generated hook environment for further usage] 2026-03-09 19:10:18.728651 | controller | ERROR 2026-03-09 19:10:18.729064 | controller | { 2026-03-09 19:10:18.729133 | controller | "ansible_facts": {}, 2026-03-09 19:10:18.729175 | controller | "ansible_included_var_files": [], 2026-03-09 19:10:18.729212 | controller | "message": "/home/zuul/ci-framework-data/artifacts directory does not exist" 2026-03-09 19:10:18.729249 | controller | } 2026-03-09 19:10:18.729295 | controller | ERROR: Ignoring Errors 2026-03-09 19:10:18.742706 | 2026-03-09 19:10:18.742887 | TASK [artifacts : Gather parameter passed to the playbook] 2026-03-09 19:10:19.478301 | controller | changed 2026-03-09 19:10:19.492062 | 2026-03-09 19:10:19.492254 | TASK [artifacts : Gather facts] 2026-03-09 19:10:20.149560 | controller | changed 2026-03-09 19:10:20.165246 | 2026-03-09 19:10:20.166078 | TASK [artifacts : Copy some network configuration files] 2026-03-09 19:10:20.748558 | controller | changed 2026-03-09 19:10:20.762795 | 2026-03-09 19:10:20.762929 | TASK [artifacts : Get installed packages list] 2026-03-09 19:10:21.810549 | controller | ok 2026-03-09 19:10:21.878366 | 2026-03-09 19:10:21.878479 | TASK [artifacts : Output package list] 2026-03-09 19:10:22.571253 | controller | changed 2026-03-09 19:10:22.588885 | 2026-03-09 19:10:22.588967 | LOOP [artifacts : Cleanup unnecesary directories] 2026-03-09 19:10:22.921180 | controller | changed: "/home/zuul/ci-framework-data/artifacts/manifests/operator" 2026-03-09 19:10:22.921804 | controller | ok: All items complete 2026-03-09 19:10:22.921875 | 2026-03-09 19:10:22.983748 | 2026-03-09 19:10:22.983951 | TASK [os_must_gather : Ensure directories are present] 2026-03-09 19:10:23.237750 | controller | changed 2026-03-09 19:10:23.262097 | 2026-03-09 19:10:23.262197 | LOOP [os_must_gather : Construct project change list] 2026-03-09 19:10:23.351438 | controller | ok: 2026-03-09 19:10:23.351846 | controller | { 2026-03-09 19:10:23.351917 | controller | "branch": "master", 2026-03-09 19:10:23.351969 | controller | "change": "979276", 2026-03-09 19:10:23.352043 | controller | "change_message": "Ensure zone migration creates the correct action\n\nFix the logic in the zone migration strategy that decides whether to\nmigrate or retype a volume. After a change in the schema[1], the zone\nmigration strategy does not support the user passing both dst_pool\nand dst_type in the same input pool.\n\nWith the current change, if dst_type is not passed, only\nmigration action should be created, never retypes.\n\n[1] https://review.opendev.org/c/openstack/watcher/+/964776\n\nChange-Id: I4c3f90eab927e4c5c902f610cb751ba0eb65001e\nSigned-off-by: jgilaber \n", 2026-03-09 19:10:23.352314 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher/+/979276", 2026-03-09 19:10:23.352376 | controller | "commit_id": "64164197e0a8ae9e5016567724b13e7a66130ae7", 2026-03-09 19:10:23.352418 | controller | "patchset": "3", 2026-03-09 19:10:23.352458 | controller | "project": { 2026-03-09 19:10:23.352496 | controller | "canonical_hostname": "opendev.org", 2026-03-09 19:10:23.352533 | controller | "canonical_name": "opendev.org/openstack/watcher", 2026-03-09 19:10:23.352569 | controller | "name": "openstack/watcher", 2026-03-09 19:10:23.352604 | controller | "short_name": "watcher", 2026-03-09 19:10:23.352639 | controller | "src_dir": "src/opendev.org/openstack/watcher" 2026-03-09 19:10:23.352674 | controller | }, 2026-03-09 19:10:23.352708 | controller | "topic": null 2026-03-09 19:10:23.352742 | controller | } 2026-03-09 19:10:23.382761 | controller | ok: 2026-03-09 19:10:23.382928 | controller | { 2026-03-09 19:10:23.382965 | controller | "branch": "master", 2026-03-09 19:10:23.382990 | controller | "change": "979643", 2026-03-09 19:10:23.383061 | controller | "change_message": "Enable SameBackendFilter in cinder scheduler\n\nAdd SameBackendFilter to cinder scheduler_default_filters\nin the zuul job configuration to control volume scheduling\nduring migration tests.\n\nChange-Id: Ie9c4391a0ab80055f31c1befeaf5383d03dc4621\nSigned-off-by: jgilaber \n", 2026-03-09 19:10:23.383103 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher/+/979643", 2026-03-09 19:10:23.383129 | controller | "commit_id": "29e4137808ab2295c733b36b9851bdace3dec60a", 2026-03-09 19:10:23.383152 | controller | "patchset": "3", 2026-03-09 19:10:23.383173 | controller | "project": { 2026-03-09 19:10:23.383194 | controller | "canonical_hostname": "opendev.org", 2026-03-09 19:10:23.383216 | controller | "canonical_name": "opendev.org/openstack/watcher", 2026-03-09 19:10:23.383237 | controller | "name": "openstack/watcher", 2026-03-09 19:10:23.383257 | controller | "short_name": "watcher", 2026-03-09 19:10:23.383278 | controller | "src_dir": "src/opendev.org/openstack/watcher" 2026-03-09 19:10:23.383299 | controller | }, 2026-03-09 19:10:23.383320 | controller | "topic": null 2026-03-09 19:10:23.383341 | controller | } 2026-03-09 19:10:23.443478 | controller | ok: 2026-03-09 19:10:23.443593 | controller | { 2026-03-09 19:10:23.443636 | controller | "branch": "master", 2026-03-09 19:10:23.443655 | controller | "change": "958644", 2026-03-09 19:10:23.443673 | controller | "change_message": "Add test for volume migrate with zone migration\n\nAdd a test that creates a zone migration audit and migrates both an\navailable and attached volume. The test checks that there is more than\none cinder pool to be able to migrate volumes between them, but makes no\nassertion on the backend configuration.\n\nDepends-On: https://review.opendev.org/c/openstack/watcher/+/979276\nDepends-On: https://review.opendev.org/c/openstack/watcher/+/979643\n\nAssited-By: Cursor (Claude-4-sonnet)\nChange-Id: I4e7e103109f830e28961856dfc3c197d89347e63\nSigned-off-by: jgilaber \n", 2026-03-09 19:10:23.443816 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/958644", 2026-03-09 19:10:23.443839 | controller | "commit_id": "5931d8d6e69fb6fa67320bbc508b3744b12bed85", 2026-03-09 19:10:23.443857 | controller | "patchset": "32", 2026-03-09 19:10:23.443872 | controller | "project": { 2026-03-09 19:10:23.443888 | controller | "canonical_hostname": "opendev.org", 2026-03-09 19:10:23.443904 | controller | "canonical_name": "opendev.org/openstack/watcher-tempest-plugin", 2026-03-09 19:10:23.443932 | controller | "name": "openstack/watcher-tempest-plugin", 2026-03-09 19:10:23.443948 | controller | "short_name": "watcher-tempest-plugin", 2026-03-09 19:10:23.443963 | controller | "src_dir": "src/opendev.org/openstack/watcher-tempest-plugin" 2026-03-09 19:10:23.443978 | controller | }, 2026-03-09 19:10:23.443992 | controller | "topic": null 2026-03-09 19:10:23.444008 | controller | } 2026-03-09 19:10:23.461109 | controller | ok: 2026-03-09 19:10:23.461161 | controller | { 2026-03-09 19:10:23.461185 | controller | "branch": "master", 2026-03-09 19:10:23.461203 | controller | "change": "962702", 2026-03-09 19:10:23.461222 | controller | "change_message": "Test zone migration volume and compute migrations\n\nReplace the volume migration test with a new one that verifies zone\nmigration strategy can migrate both volumes and compute instances\nin the same action plan.\n\nGenerated-By: Cursor (claude-4-sonnet)\nChange-Id: I748d1316356e6ad116791eb33c3451c2a9c41636\nSigned-off-by: jgilaber \n", 2026-03-09 19:10:23.461242 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/962702", 2026-03-09 19:10:23.461258 | controller | "commit_id": "b0c5f8ac8fc6c508dc18eab2816d38f4c959083b", 2026-03-09 19:10:23.461274 | controller | "patchset": "25", 2026-03-09 19:10:23.461290 | controller | "project": { 2026-03-09 19:10:23.461305 | controller | "canonical_hostname": "opendev.org", 2026-03-09 19:10:23.461321 | controller | "canonical_name": "opendev.org/openstack/watcher-tempest-plugin", 2026-03-09 19:10:23.461336 | controller | "name": "openstack/watcher-tempest-plugin", 2026-03-09 19:10:23.461351 | controller | "short_name": "watcher-tempest-plugin", 2026-03-09 19:10:23.461366 | controller | "src_dir": "src/opendev.org/openstack/watcher-tempest-plugin" 2026-03-09 19:10:23.461380 | controller | }, 2026-03-09 19:10:23.461395 | controller | "topic": null 2026-03-09 19:10:23.461410 | controller | } 2026-03-09 19:10:23.478003 | controller | ok: 2026-03-09 19:10:23.478068 | controller | { 2026-03-09 19:10:23.478090 | controller | "branch": "master", 2026-03-09 19:10:23.478107 | controller | "change": "963559", 2026-03-09 19:10:23.478123 | controller | "change_message": "Add extra checks to zone migration retype test\n\nAdd two extra checks to the zone migration test covering volume retype\nto prevent regressions of two bugs:\n\n1. Ensure the src_type parameter filter volumes to retype[1]\n2. Ensure that the audit does not fail if the 'with_attached_volume'\n parameter is passed to an audit without 'compute_nodes'[2]\n\n[1] https://bugs.launchpad.net/watcher/+bug/2111507\n[2] https://bugs.launchpad.net/watcher/+bug/2111429\n\nChange-Id: I3987230d384b509921ded80777b4d5b7c407ba85\nSigned-off-by: jgilaber \n", 2026-03-09 19:10:23.478140 | controller | "change_url": "https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559", 2026-03-09 19:10:23.478156 | controller | "commit_id": "6fedf196e61ec8ee8c9655e8fa463f714bd0ce13", 2026-03-09 19:10:23.478172 | controller | "patchset": "16", 2026-03-09 19:10:23.478187 | controller | "project": { 2026-03-09 19:10:23.478203 | controller | "canonical_hostname": "opendev.org", 2026-03-09 19:10:23.478218 | controller | "canonical_name": "opendev.org/openstack/watcher-tempest-plugin", 2026-03-09 19:10:23.478233 | controller | "name": "openstack/watcher-tempest-plugin", 2026-03-09 19:10:23.478248 | controller | "short_name": "watcher-tempest-plugin", 2026-03-09 19:10:23.478262 | controller | "src_dir": "src/opendev.org/openstack/watcher-tempest-plugin" 2026-03-09 19:10:23.478277 | controller | }, 2026-03-09 19:10:23.478291 | controller | "topic": null 2026-03-09 19:10:23.478306 | controller | } 2026-03-09 19:10:23.494972 | controller | ok: 2026-03-09 19:10:23.495079 | controller | { 2026-03-09 19:10:23.495108 | controller | "branch": "main", 2026-03-09 19:10:23.495140 | controller | "change": "351", 2026-03-09 19:10:23.495270 | controller | "change_message": "Enable SameBackendFilter in cinder scheduler\n\nAdd SameBackendFilter to cinder scheduler_default_filters\nin the nfs job configuration to control volume scheduling\nduring migration tests.\n\nDepends-On: https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559\n", 2026-03-09 19:10:23.495295 | controller | "change_url": "https://github.com/openstack-k8s-operators/watcher-operator/pull/351", 2026-03-09 19:10:23.495313 | controller | "commit_id": "2e035aad6e396aeb72cc6aec8684c43e59f8b674", 2026-03-09 19:10:23.495329 | controller | "patchset": "2e035aad6e396aeb72cc6aec8684c43e59f8b674", 2026-03-09 19:10:23.495345 | controller | "project": { 2026-03-09 19:10:23.495362 | controller | "canonical_hostname": "github.com", 2026-03-09 19:10:23.495377 | controller | "canonical_name": "github.com/openstack-k8s-operators/watcher-operator", 2026-03-09 19:10:23.495393 | controller | "name": "openstack-k8s-operators/watcher-operator", 2026-03-09 19:10:23.495408 | controller | "short_name": "watcher-operator", 2026-03-09 19:10:23.495423 | controller | "src_dir": "src/github.com/openstack-k8s-operators/watcher-operator" 2026-03-09 19:10:23.495438 | controller | }, 2026-03-09 19:10:23.495454 | controller | "topic": null 2026-03-09 19:10:23.495469 | controller | } 2026-03-09 19:10:23.604662 | 2026-03-09 19:10:23.604792 | TASK [os_must_gather : Get git tag for image tagging] 2026-03-09 19:10:24.224952 | controller | skipping: Conditional result was False 2026-03-09 19:10:24.267812 | 2026-03-09 19:10:24.268074 | TASK [os_must_gather : Debug git show-ref] 2026-03-09 19:10:24.306146 | controller | skipping: Conditional result was False 2026-03-09 19:10:24.350528 | 2026-03-09 19:10:24.350671 | TASK [os_must_gather : Build openstack-must-gather container] 2026-03-09 19:10:24.387338 | controller | skipping: Conditional result was False 2026-03-09 19:10:24.425616 | 2026-03-09 19:10:24.425763 | TASK [os_must_gather : Push openstack-must-gather container] 2026-03-09 19:10:24.462089 | controller | skipping: Conditional result was False 2026-03-09 19:10:24.531387 | 2026-03-09 19:10:24.531514 | TASK [os_must_gather : Set cifmw_os_must_gather_image to new image tag] 2026-03-09 19:10:24.566800 | controller | skipping: Conditional result was False 2026-03-09 19:10:24.587308 | 2026-03-09 19:10:24.587401 | TASK [os_must_gather : Check for oc command] 2026-03-09 19:10:25.033859 | controller | changed 2026-03-09 19:10:25.062518 | 2026-03-09 19:10:25.062675 | TASK [os_must_gather : Check if kubeconfig exists] 2026-03-09 19:10:25.362639 | controller | ok 2026-03-09 19:10:25.391986 | 2026-03-09 19:10:25.392121 | TASK [os_must_gather : Run openstack-must-gather command] 2026-03-09 19:15:39.725142 | controller | changed 2026-03-09 19:15:39.757565 | 2026-03-09 19:15:39.757678 | TASK [os_must_gather : Find existing os-must-gather directories] 2026-03-09 19:15:40.012696 | controller | ok: All paths examined 2026-03-09 19:15:40.043826 | 2026-03-09 19:15:40.043998 | TASK [os_must_gather : Create a symlink to newest os-must-gather directory] 2026-03-09 19:15:40.319950 | controller | changed 2026-03-09 19:15:40.349749 | 2026-03-09 19:15:40.349884 | TASK [os_must_gather : Collect pod usage] 2026-03-09 19:15:40.422676 | controller | ok 2026-03-09 19:15:40.450771 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/os_must_gather/tasks/get_top.yml 2026-03-09 19:15:40.472990 | 2026-03-09 19:15:40.473087 | TASK [os_must_gather : Get all containers usage - sort by cpu] 2026-03-09 19:15:40.928110 | controller | error: Metrics API not available 2026-03-09 19:15:41.135894 | controller | ERROR 2026-03-09 19:15:41.136210 | controller | { 2026-03-09 19:15:41.136301 | controller | "delta": "0:00:00.096297", 2026-03-09 19:15:41.136350 | controller | "end": "2026-03-09 19:15:40.930851", 2026-03-09 19:15:41.136419 | controller | "msg": "non-zero return code", 2026-03-09 19:15:41.136464 | controller | "rc": 1, 2026-03-09 19:15:41.136509 | controller | "start": "2026-03-09 19:15:40.834554" 2026-03-09 19:15:41.136546 | controller | } 2026-03-09 19:15:41.136602 | controller | ERROR: Ignoring Errors 2026-03-09 19:15:41.180135 | 2026-03-09 19:15:41.180356 | TASK [os_must_gather : Print message when metrics are not available] 2026-03-09 19:15:41.219351 | controller | skipping: Conditional result was False 2026-03-09 19:15:41.253282 | 2026-03-09 19:15:41.253447 | TASK [os_must_gather : Get all containers usage - sort by memory] 2026-03-09 19:15:41.891856 | controller | skipping: Conditional result was False 2026-03-09 19:15:41.935693 | 2026-03-09 19:15:41.935853 | TASK [os_must_gather : Find all namespaces directories] 2026-03-09 19:15:41.973830 | controller | skipping: Conditional result was False 2026-03-09 19:15:42.003142 | 2026-03-09 19:15:42.003245 | LOOP [os_must_gather : Get resource usage by pods per namespace] 2026-03-09 19:15:42.077554 | 2026-03-09 19:15:42.077737 | TASK [os_must_gather : Get node resource usage] 2026-03-09 19:15:42.748835 | controller | skipping: Conditional result was False 2026-03-09 19:15:42.819086 | 2026-03-09 19:15:42.819211 | TASK [artifacts : Create crc logs directory] 2026-03-09 19:15:43.086509 | controller | changed 2026-03-09 19:15:43.110702 | 2026-03-09 19:15:43.110803 | TASK [artifacts : Ensure controller knows CRC ssh keys] 2026-03-09 19:15:43.779252 | controller | changed 2026-03-09 19:15:43.820359 | 2026-03-09 19:15:43.820528 | TASK [artifacts : Recognize new keypair] 2026-03-09 19:15:44.063217 | controller | ok 2026-03-09 19:15:44.090551 | 2026-03-09 19:15:44.090656 | TASK [artifacts : Set fact if new keypair exists] 2026-03-09 19:15:44.128211 | controller | skipping: Conditional result was False 2026-03-09 19:15:44.156517 | 2026-03-09 19:15:44.156623 | TASK [artifacts : Prepare root ssh accesses] 2026-03-09 19:15:48.218318 | controller | changed 2026-03-09 19:15:48.254911 | 2026-03-09 19:15:48.255076 | TASK [artifacts : Copy logs from CRC VM] 2026-03-09 19:15:50.888846 | controller | changed 2026-03-09 19:15:50.923235 | 2026-03-09 19:15:50.923382 | TASK [artifacts : List all of the existing virtual machines] 2026-03-09 19:15:50.959435 | controller | skipping: Conditional result was False 2026-03-09 19:15:51.039363 | 2026-03-09 19:15:51.039529 | TASK [artifacts : Filter out edpm vm] 2026-03-09 19:15:51.075153 | controller | skipping: Conditional result was False 2026-03-09 19:15:51.095508 | 2026-03-09 19:15:51.095696 | TASK [artifacts : Slurp zuul inventory file] 2026-03-09 19:15:51.368722 | controller | ok 2026-03-09 19:15:51.396956 | 2026-03-09 19:15:51.397119 | LOOP [artifacts : Extract Compute and Networker from zuul mapping if any] 2026-03-09 19:15:51.511289 | 2026-03-09 19:15:51.511510 | LOOP [artifacts : Generate logs on edpm vm {{ item.split('/')[0] }}] 2026-03-09 19:15:51.581706 | 2026-03-09 19:15:51.581918 | LOOP [artifacts : Copy logs to host machine from {{ item.split('/')[0] }}] 2026-03-09 19:15:51.648366 | 2026-03-09 19:15:51.648590 | TASK [artifacts : Generate list of ansible logs to collect in home directory] 2026-03-09 19:15:51.891904 | controller | ok: All paths examined 2026-03-09 19:15:51.919312 | 2026-03-09 19:15:51.919428 | LOOP [artifacts : Copy ansible logs to logs directory] 2026-03-09 19:15:52.365845 | controller | changed: 2026-03-09 19:15:52.366011 | controller | { 2026-03-09 19:15:52.366058 | controller | "atime": 1773081157.9173064, 2026-03-09 19:15:52.366077 | controller | "ctime": 1773083403.9601207, 2026-03-09 19:15:52.366095 | controller | "dev": 64513, 2026-03-09 19:15:52.366112 | controller | "gid": 1000, 2026-03-09 19:15:52.366129 | controller | "gr_name": "zuul", 2026-03-09 19:15:52.366145 | controller | "inode": 1964301, 2026-03-09 19:15:52.366161 | controller | "isblk": false, 2026-03-09 19:15:52.366177 | controller | "ischr": false, 2026-03-09 19:15:52.366192 | controller | "isdir": false, 2026-03-09 19:15:52.366215 | controller | "isfifo": false, 2026-03-09 19:15:52.366232 | controller | "isgid": false, 2026-03-09 19:15:52.366248 | controller | "islnk": false, 2026-03-09 19:15:52.366263 | controller | "isreg": true, 2026-03-09 19:15:52.366279 | controller | "issock": false, 2026-03-09 19:15:52.366295 | controller | "isuid": false, 2026-03-09 19:15:52.366310 | controller | "mode": "0644", 2026-03-09 19:15:52.366327 | controller | "mtime": 1773083403.9601207, 2026-03-09 19:15:52.366342 | controller | "nlink": 1, 2026-03-09 19:15:52.366358 | controller | "path": "/home/zuul/ansible.log", 2026-03-09 19:15:52.366373 | controller | "pw_name": "zuul", 2026-03-09 19:15:52.366389 | controller | "rgrp": true, 2026-03-09 19:15:52.366404 | controller | "roth": true, 2026-03-09 19:15:52.366420 | controller | "rusr": true, 2026-03-09 19:15:52.366435 | controller | "size": 6314987, 2026-03-09 19:15:52.366451 | controller | "uid": 1000, 2026-03-09 19:15:52.366467 | controller | "wgrp": false, 2026-03-09 19:15:52.366482 | controller | "woth": false, 2026-03-09 19:15:52.366497 | controller | "wusr": true, 2026-03-09 19:15:52.366512 | controller | "xgrp": false, 2026-03-09 19:15:52.366528 | controller | "xoth": false, 2026-03-09 19:15:52.366543 | controller | "xusr": false 2026-03-09 19:15:52.366557 | controller | } 2026-03-09 19:15:52.366583 | 2026-03-09 19:15:52.407395 | 2026-03-09 19:15:52.407526 | TASK [artifacts : Ensure we have proper rights on the gathered content] 2026-03-09 19:15:54.591193 | controller | changed 2026-03-09 19:15:54.631612 | 2026-03-09 19:15:54.631796 | LOOP [artifacts : Mask secrets in yaml log files] 2026-03-09 19:18:46.365656 | controller | changed: "/home/zuul/ci-framework-data/logs" 2026-03-09 19:18:47.126148 | controller | changed: "/home/zuul/ci-framework-data/artifacts" 2026-03-09 19:18:47.192122 | 2026-03-09 19:18:47.192254 | LOOP [env_op_images : Ensure directory is present] 2026-03-09 19:18:47.443990 | controller | ok: "artifacts" 2026-03-09 19:18:47.444388 | controller | changed: All items complete 2026-03-09 19:18:47.444443 | 2026-03-09 19:18:47.676725 | controller | ok: "logs" 2026-03-09 19:18:47.706659 | 2026-03-09 19:18:47.706786 | TASK [env_op_images : Check if OpenStackControlPlane is setup] 2026-03-09 19:18:48.330875 | controller | changed 2026-03-09 19:18:48.375254 | 2026-03-09 19:18:48.375473 | TASK [env_op_images : Get images from the CSV] 2026-03-09 19:18:49.093084 | controller | skipping: Conditional result was False 2026-03-09 19:18:49.134933 | 2026-03-09 19:18:49.135171 | TASK [env_op_images : Get the images name] 2026-03-09 19:18:49.766949 | controller | skipping: Conditional result was False 2026-03-09 19:18:49.808572 | 2026-03-09 19:18:49.808892 | TASK [env_op_images : Extract env variable name and images] 2026-03-09 19:18:49.846670 | controller | skipping: Conditional result was False 2026-03-09 19:18:49.888496 | 2026-03-09 19:18:49.888621 | TASK [env_op_images : Get all pods from all namespaces to find openstack-operator-index] 2026-03-09 19:18:49.926897 | controller | skipping: Conditional result was False 2026-03-09 19:18:49.969198 | 2026-03-09 19:18:49.969318 | TASK [env_op_images : Retrieve openstack-operator-index pod] 2026-03-09 19:18:50.005768 | controller | skipping: Conditional result was False 2026-03-09 19:18:50.045338 | 2026-03-09 19:18:50.045482 | TASK [env_op_images : Get all the pods in openstack-operator namespace] 2026-03-09 19:18:50.082289 | controller | skipping: Conditional result was False 2026-03-09 19:18:50.166995 | 2026-03-09 19:18:50.167146 | TASK [env_op_images : Get operator images and pods] 2026-03-09 19:18:50.212832 | controller | skipping: Conditional result was False 2026-03-09 19:18:50.249071 | 2026-03-09 19:18:50.249171 | LOOP [env_op_images : Add operator images to the dictionary] 2026-03-09 19:18:50.322696 | 2026-03-09 19:18:50.322920 | TASK [env_op_images : Write images to file] 2026-03-09 19:18:50.359208 | controller | skipping: Conditional result was False 2026-03-09 19:18:50.419767 | 2026-03-09 19:18:50.419904 | TASK [cifmw_setup : Create a versioned log folder] 2026-03-09 19:18:50.679524 | controller | changed 2026-03-09 19:18:50.706397 | 2026-03-09 19:18:50.706485 | TASK [cifmw_setup : Return a list of log files in home directory] 2026-03-09 19:18:50.958198 | controller | ok: All paths examined 2026-03-09 19:18:50.987727 | 2026-03-09 19:18:50.987825 | TASK [cifmw_setup : Ensure ansible facts cache exists] 2026-03-09 19:18:51.233720 | controller | ok 2026-03-09 19:18:51.260404 | 2026-03-09 19:18:51.260517 | LOOP [cifmw_setup : Copy logs to proper location] 2026-03-09 19:18:51.658709 | controller | changed: 2026-03-09 19:18:51.658983 | controller | { 2026-03-09 19:18:51.659072 | controller | "atime": 1773083752.2550323, 2026-03-09 19:18:51.659117 | controller | "ctime": 1773083403.9601207, 2026-03-09 19:18:51.659154 | controller | "dev": 64513, 2026-03-09 19:18:51.659192 | controller | "gid": 1000, 2026-03-09 19:18:51.659228 | controller | "gr_name": "zuul", 2026-03-09 19:18:51.659265 | controller | "inode": 1964301, 2026-03-09 19:18:51.659300 | controller | "isblk": false, 2026-03-09 19:18:51.659335 | controller | "ischr": false, 2026-03-09 19:18:51.659370 | controller | "isdir": false, 2026-03-09 19:18:51.659403 | controller | "isfifo": false, 2026-03-09 19:18:51.659437 | controller | "isgid": false, 2026-03-09 19:18:51.659472 | controller | "islnk": false, 2026-03-09 19:18:51.659506 | controller | "isreg": true, 2026-03-09 19:18:51.659541 | controller | "issock": false, 2026-03-09 19:18:51.659574 | controller | "isuid": false, 2026-03-09 19:18:51.659608 | controller | "mode": "0644", 2026-03-09 19:18:51.659641 | controller | "mtime": 1773083403.9601207, 2026-03-09 19:18:51.659676 | controller | "nlink": 1, 2026-03-09 19:18:51.659717 | controller | "path": "/home/zuul/ansible.log", 2026-03-09 19:18:51.659754 | controller | "pw_name": "zuul", 2026-03-09 19:18:51.659791 | controller | "rgrp": true, 2026-03-09 19:18:51.659827 | controller | "roth": true, 2026-03-09 19:18:51.659863 | controller | "rusr": true, 2026-03-09 19:18:51.659898 | controller | "size": 6314987, 2026-03-09 19:18:51.659932 | controller | "uid": 1000, 2026-03-09 19:18:51.659966 | controller | "wgrp": false, 2026-03-09 19:18:51.659999 | controller | "woth": false, 2026-03-09 19:18:51.660071 | controller | "wusr": true, 2026-03-09 19:18:51.660108 | controller | "xgrp": false, 2026-03-09 19:18:51.660142 | controller | "xoth": false, 2026-03-09 19:18:51.660175 | controller | "xusr": false 2026-03-09 19:18:51.660208 | controller | } 2026-03-09 19:18:51.660257 | 2026-03-09 19:18:51.688999 | 2026-03-09 19:18:51.689111 | LOOP [cifmw_setup : Remove original log from home directory] 2026-03-09 19:18:51.934777 | controller | changed: 2026-03-09 19:18:51.934951 | controller | { 2026-03-09 19:18:51.934984 | controller | "atime": 1773083752.2550323, 2026-03-09 19:18:51.935003 | controller | "ctime": 1773083403.9601207, 2026-03-09 19:18:51.935046 | controller | "dev": 64513, 2026-03-09 19:18:51.935069 | controller | "gid": 1000, 2026-03-09 19:18:51.935091 | controller | "gr_name": "zuul", 2026-03-09 19:18:51.935118 | controller | "inode": 1964301, 2026-03-09 19:18:51.935139 | controller | "isblk": false, 2026-03-09 19:18:51.935154 | controller | "ischr": false, 2026-03-09 19:18:51.935176 | controller | "isdir": false, 2026-03-09 19:18:51.935196 | controller | "isfifo": false, 2026-03-09 19:18:51.935217 | controller | "isgid": false, 2026-03-09 19:18:51.935259 | controller | "islnk": false, 2026-03-09 19:18:51.935335 | controller | "isreg": true, 2026-03-09 19:18:51.935364 | controller | "issock": false, 2026-03-09 19:18:51.935388 | controller | "isuid": false, 2026-03-09 19:18:51.935410 | controller | "mode": "0644", 2026-03-09 19:18:51.935430 | controller | "mtime": 1773083403.9601207, 2026-03-09 19:18:51.935548 | controller | "nlink": 1, 2026-03-09 19:18:51.935571 | controller | "path": "/home/zuul/ansible.log", 2026-03-09 19:18:51.935587 | controller | "pw_name": "zuul", 2026-03-09 19:18:51.935611 | controller | "rgrp": true, 2026-03-09 19:18:51.935626 | controller | "roth": true, 2026-03-09 19:18:51.935641 | controller | "rusr": true, 2026-03-09 19:18:51.935656 | controller | "size": 6314987, 2026-03-09 19:18:51.935671 | controller | "uid": 1000, 2026-03-09 19:18:51.935687 | controller | "wgrp": false, 2026-03-09 19:18:51.935702 | controller | "woth": false, 2026-03-09 19:18:51.935716 | controller | "wusr": true, 2026-03-09 19:18:51.935730 | controller | "xgrp": false, 2026-03-09 19:18:51.935745 | controller | "xoth": false, 2026-03-09 19:18:51.935759 | controller | "xusr": false 2026-03-09 19:18:51.935774 | controller | } 2026-03-09 19:18:52.018354 | 2026-03-09 19:18:52.018487 | TASK [cifmw_setup : Copy facts to dated directory] 2026-03-09 19:18:52.432685 | controller | changed 2026-03-09 19:18:52.465452 | 2026-03-09 19:18:52.465607 | TASK [run_hook : Assert parameters are valid] 2026-03-09 19:18:52.624465 | controller | ok: All assertions passed 2026-03-09 19:18:52.652273 | 2026-03-09 19:18:52.652432 | TASK [run_hook : Assert single hooks are all mappings] 2026-03-09 19:18:52.794492 | controller | ok: All assertions passed 2026-03-09 19:18:52.823331 | 2026-03-09 19:18:52.823446 | LOOP [run_hook : Loop on hooks for post_logs] 2026-03-09 19:18:53.113161 | 2026-03-09 19:18:53.113324 | PLAY [Run ci/playbooks/e2e-collect-logs.yml on CRC host] 2026-03-09 19:18:53.132557 | 2026-03-09 19:18:53.132624 | TASK [Get kubelet journalctl logs] 2026-03-09 19:18:54.225991 | crc | Output suppressed because no_log was given 2026-03-09 19:18:54.235063 | 2026-03-09 19:18:54.235205 | TASK [Compress logs bigger than 2MB] 2026-03-09 19:18:54.836752 | crc | changed 2026-03-09 19:18:54.851722 | 2026-03-09 19:18:54.851861 | TASK [Copy files from workspace on node] 2026-03-09 19:18:54.893186 | crc | ok 2026-03-09 19:18:54.935735 | 2026-03-09 19:18:54.935853 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-09 19:18:54.998370 | crc | ok 2026-03-09 19:18:55.012538 | 2026-03-09 19:18:55.012673 | TASK [fetch-output : Set log path for single node] 2026-03-09 19:18:55.049194 | crc | skipping: Conditional result was False 2026-03-09 19:18:55.065404 | 2026-03-09 19:18:55.065545 | LOOP [fetch-output : Ensure local output dirs] 2026-03-09 19:18:55.344311 | crc -> localhost | changed: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/logs/crc" 2026-03-09 19:18:55.344795 | 2026-03-09 19:18:55.563045 | crc -> localhost | changed: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/artifacts" 2026-03-09 19:18:55.791565 | crc -> localhost | changed: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/docs" 2026-03-09 19:18:55.804923 | 2026-03-09 19:18:55.805068 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-09 19:18:56.598228 | crc | changed: 2026-03-09 19:18:56.598578 | crc | .d..t...... ./ 2026-03-09 19:18:56.598640 | crc | >f+++++++++ kubelet.log.gz 2026-03-09 19:18:57.160705 | crc | changed: .d..t...... ./ 2026-03-09 19:18:57.729183 | crc | changed: .d..t...... ./ 2026-03-09 19:18:57.794922 | 2026-03-09 19:18:57.795088 | PLAY RECAP 2026-03-09 19:18:57.795149 | controller | ok: 75 changed: 34 unreachable: 0 failed: 0 skipped: 39 rescued: 0 ignored: 2 2026-03-09 19:18:57.795191 | crc | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0 2026-03-09 19:18:57.795216 | 2026-03-09 19:18:57.974390 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-collect-logs.yml@main] 2026-03-09 19:18:57.984308 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml@main] 2026-03-09 19:18:58.594424 | 2026-03-09 19:18:58.594542 | PLAY [Run ci/playbooks/collect-logs.yml] 2026-03-09 19:18:58.707231 | 2026-03-09 19:18:58.707385 | TASK [Filter out host if needed] 2026-03-09 19:18:58.737668 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller 2026-03-09 19:18:58.744458 | 2026-03-09 19:18:58.744574 | TASK [Filter out host if needed] 2026-03-09 19:18:58.779090 | 2026-03-09 19:18:58.779250 | TASK [Ensure file is present] 2026-03-09 19:18:59.286546 | controller | ok 2026-03-09 19:18:59.319151 | 2026-03-09 19:18:59.319283 | TASK [Manage molecule report file] 2026-03-09 19:19:00.011752 | controller | skipping: Conditional result was False 2026-03-09 19:19:00.064156 | 2026-03-09 19:19:00.064293 | TASK [Check if we get ci-framework-data basedir] 2026-03-09 19:19:00.337068 | controller | ok 2026-03-09 19:19:00.398939 | 2026-03-09 19:19:00.399090 | TASK [Create ci-framework-data log directory for zuul] 2026-03-09 19:19:00.836196 | controller | changed 2026-03-09 19:19:00.872517 | 2026-03-09 19:19:00.872650 | TASK [Copy ci-framework interesting files] 2026-03-09 19:19:01.575595 | controller | cp: cannot stat '/home/zuul/ci-framework-data/tests': No such file or directory 2026-03-09 19:19:02.073099 | controller | changed 2026-03-09 19:19:02.125546 | 2026-03-09 19:19:02.125737 | TASK [Get SELinux listing] 2026-03-09 19:19:02.819437 | controller | changed 2026-03-09 19:19:02.871965 | 2026-03-09 19:19:02.872107 | TASK [Generate log index] 2026-03-09 19:19:03.695189 | controller | changed 2026-03-09 19:19:03.726833 | 2026-03-09 19:19:03.726948 | TASK [Get some env related data] 2026-03-09 19:19:04.473234 | controller | /home/zuul/.local/bin/ansible 2026-03-09 19:19:05.409348 | controller | changed 2026-03-09 19:19:05.466046 | 2026-03-09 19:19:05.466177 | TASK [Generate list of logs to collect in home directory] 2026-03-09 19:19:05.838676 | controller | ok: All paths examined 2026-03-09 19:19:05.869936 | 2026-03-09 19:19:05.870112 | LOOP [Copy logs from home directory] 2026-03-09 19:19:05.920633 | 2026-03-09 19:19:05.920836 | TASK [Copy crio stats log file] 2026-03-09 19:19:05.955263 | controller | skipping: Conditional result was False 2026-03-09 19:19:05.979557 | 2026-03-09 19:19:05.979643 | TASK [Get SELinux related data] 2026-03-09 19:19:06.373829 | controller | 2026-03-09 19:19:06.673306 | controller | ERROR 2026-03-09 19:19:06.673684 | controller | { 2026-03-09 19:19:06.673747 | controller | "delta": "0:00:00.008865", 2026-03-09 19:19:06.673789 | controller | "end": "2026-03-09 19:19:06.374519", 2026-03-09 19:19:06.673828 | controller | "msg": "non-zero return code", 2026-03-09 19:19:06.673864 | controller | "rc": 1, 2026-03-09 19:19:06.673900 | controller | "start": "2026-03-09 19:19:06.365654" 2026-03-09 19:19:06.673933 | controller | } 2026-03-09 19:19:06.673979 | controller | ERROR: Ignoring Errors 2026-03-09 19:19:06.725605 | 2026-03-09 19:19:06.725756 | TASK [Create system configuration directory] 2026-03-09 19:19:07.037984 | controller | changed 2026-03-09 19:19:07.069106 | 2026-03-09 19:19:07.069176 | TASK [Get some of the system configurations] 2026-03-09 19:19:07.749895 | controller | changed 2026-03-09 19:19:07.804748 | 2026-03-09 19:19:07.804913 | TASK [Copy generated documentation if available] 2026-03-09 19:19:07.832447 | controller | skipping: Conditional result was False 2026-03-09 19:19:07.867644 | 2026-03-09 19:19:07.867787 | TASK [Copy generated AsciiDoc documentation if available] 2026-03-09 19:19:07.896831 | controller | skipping: Conditional result was False 2026-03-09 19:19:07.932119 | 2026-03-09 19:19:07.932251 | TASK [Compress logs bigger than 2MB] 2026-03-09 19:19:26.744218 | controller | changed 2026-03-09 19:19:26.795792 | 2026-03-09 19:19:26.795937 | TASK [Copy files from workspace on node] 2026-03-09 19:19:26.854599 | controller | ok 2026-03-09 19:19:26.924264 | 2026-03-09 19:19:26.924394 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-09 19:19:26.985802 | controller | ok 2026-03-09 19:19:27.047229 | 2026-03-09 19:19:27.047381 | TASK [fetch-output : Set log path for single node] 2026-03-09 19:19:27.073533 | controller | skipping: Conditional result was False 2026-03-09 19:19:27.100414 | 2026-03-09 19:19:27.100519 | LOOP [fetch-output : Ensure local output dirs] 2026-03-09 19:19:27.380157 | controller -> localhost | changed: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/logs/controller" 2026-03-09 19:19:27.380630 | 2026-03-09 19:19:27.624404 | controller -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/artifacts" 2026-03-09 19:19:27.848150 | controller -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/docs" 2026-03-09 19:19:27.904879 | 2026-03-09 19:19:27.905063 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-09 19:19:31.978978 | controller | changed: 2026-03-09 19:19:31.979316 | controller | .d..t...... ./ 2026-03-09 19:19:31.979360 | controller | >f+++++++++ README.html 2026-03-09 19:19:31.979390 | controller | >f+++++++++ dmesg.log 2026-03-09 19:19:31.979418 | controller | >f+++++++++ installed-pkgs.log 2026-03-09 19:19:31.979444 | controller | >f+++++++++ python.log 2026-03-09 19:19:31.979469 | controller | >f+++++++++ registries.conf 2026-03-09 19:19:31.979494 | controller | >f+++++++++ selinux-denials.log 2026-03-09 19:19:31.979519 | controller | >f+++++++++ selinux-listing.log 2026-03-09 19:19:31.979545 | controller | cd+++++++++ ci-framework-data/ 2026-03-09 19:19:31.979586 | controller | cd+++++++++ ci-framework-data/artifacts/ 2026-03-09 19:19:31.979611 | controller | >f+++++++++ ci-framework-data/artifacts/ansible-facts.yml 2026-03-09 19:19:31.979640 | controller | >f+++++++++ ci-framework-data/artifacts/ansible-vars.yml 2026-03-09 19:19:31.979664 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_check_for_oc.sh 2026-03-09 19:19:31.979688 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_copy_logs_from_crc.sh 2026-03-09 19:19:31.979713 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_prepare_root_ssh.sh 2026-03-09 19:19:31.979737 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_run_download.sh 2026-03-09 19:19:31.979760 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_run_openstack_must_gather.sh 2026-03-09 19:19:31.979783 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_001_run_hook_without_retry.sh 2026-03-09 19:19:31.979806 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_002_fetch_openshift.sh 2026-03-09 19:19:31.979830 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_003_run_openstack.sh 2026-03-09 19:19:31.979853 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_004_run_hook_without_retry_deploy.sh 2026-03-09 19:19:31.979875 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_005_stop_watcher.sh 2026-03-09 19:19:31.979898 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_006_install_watcher_operator_in.sh 2026-03-09 19:19:31.979921 | controller | >f+++++++++ ci-framework-data/artifacts/hosts 2026-03-09 19:19:31.979945 | controller | >f+++++++++ ci-framework-data/artifacts/install_yamls.sh 2026-03-09 19:19:31.979968 | controller | >f+++++++++ ci-framework-data/artifacts/installed-packages.yml 2026-03-09 19:19:31.979990 | controller | >f+++++++++ ci-framework-data/artifacts/ip-network.txt 2026-03-09 19:19:31.980040 | controller | >f+++++++++ ci-framework-data/artifacts/resolv.conf 2026-03-09 19:19:31.980075 | controller | >f+++++++++ ci-framework-data/artifacts/zuul_inventory.yml 2026-03-09 19:19:31.980102 | controller | cd+++++++++ ci-framework-data/artifacts/NetworkManager/ 2026-03-09 19:19:31.980126 | controller | >f+++++++++ ci-framework-data/artifacts/NetworkManager/ci-private-network.nmconnection 2026-03-09 19:19:31.980149 | controller | >f+++++++++ ci-framework-data/artifacts/NetworkManager/ens3.nmconnection 2026-03-09 19:19:31.980173 | controller | cd+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-09_19-18/ 2026-03-09 19:19:31.980196 | controller | cd+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-09_19-18/ansible_facts_cache/ 2026-03-09 19:19:31.980219 | controller | >f+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-09_19-18/ansible_facts_cache/controller 2026-03-09 19:19:31.980256 | controller | >f+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-09_19-18/ansible_facts_cache/localhost 2026-03-09 19:19:31.980279 | controller | cd+++++++++ ci-framework-data/artifacts/ci-env/ 2026-03-09 19:19:31.980303 | controller | >f+++++++++ ci-framework-data/artifacts/ci-env/networking-info.yml 2026-03-09 19:19:31.980326 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/ 2026-03-09 19:19:31.980353 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/ 2026-03-09 19:19:31.980377 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/namespace.yaml 2026-03-09 19:19:31.980400 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/ 2026-03-09 19:19:31.980426 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/ 2026-03-09 19:19:31.980451 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/operatorgroup.yaml 2026-03-09 19:19:31.980476 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/subscription.yaml 2026-03-09 19:19:31.980500 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/ 2026-03-09 19:19:31.980525 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/namespace.yaml 2026-03-09 19:19:31.980549 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/ 2026-03-09 19:19:31.980573 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/ 2026-03-09 19:19:31.980596 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgpadvertisement.yaml 2026-03-09 19:19:31.980619 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgpextras.yaml 2026-03-09 19:19:31.980642 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgppeers.yaml 2026-03-09 19:19:31.980665 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/deploy_operator.yaml 2026-03-09 19:19:31.980687 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/ipaddresspools.yaml 2026-03-09 19:19:31.980710 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/l2advertisement.yaml 2026-03-09 19:19:31.980733 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/ 2026-03-09 19:19:31.980757 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/namespace.yaml 2026-03-09 19:19:31.980780 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/ 2026-03-09 19:19:31.980804 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/cr/ 2026-03-09 19:19:31.980827 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/cr/deploy_operator.yaml 2026-03-09 19:19:31.980850 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/ 2026-03-09 19:19:31.980873 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/namespace.yaml 2026-03-09 19:19:31.980896 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/ 2026-03-09 19:19:31.980919 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/ 2026-03-09 19:19:31.980941 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/operatorgroup.yaml 2026-03-09 19:19:31.980964 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/subscription.yaml 2026-03-09 19:19:31.981116 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/ 2026-03-09 19:19:31.981145 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/ 2026-03-09 19:19:31.981175 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/operatorgroup.yaml 2026-03-09 19:19:31.981199 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/subscription.yaml 2026-03-09 19:19:31.981221 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/ 2026-03-09 19:19:31.981243 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/ 2026-03-09 19:19:31.981265 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/catalogsource.yaml 2026-03-09 19:19:31.981287 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/operatorgroup.yaml 2026-03-09 19:19:31.981308 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/subscription.yaml 2026-03-09 19:19:31.981330 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/ 2026-03-09 19:19:31.981352 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/namespace.yaml 2026-03-09 19:19:31.981374 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/cr/ 2026-03-09 19:19:31.981396 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/ 2026-03-09 19:19:31.981418 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/ 2026-03-09 19:19:31.981440 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/ctlplane.yaml 2026-03-09 19:19:31.981462 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/datacentre.yaml 2026-03-09 19:19:31.981484 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/designate.yaml 2026-03-09 19:19:31.981506 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/designateext.yaml 2026-03-09 19:19:31.981528 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/internalapi.yaml 2026-03-09 19:19:31.981550 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/octavia.yaml 2026-03-09 19:19:31.981571 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/storage.yaml 2026-03-09 19:19:31.981593 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/storagemgmt.yaml 2026-03-09 19:19:31.981614 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/tenant.yaml 2026-03-09 19:19:31.981636 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/ 2026-03-09 19:19:31.981659 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/cr/ 2026-03-09 19:19:31.981681 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/cr/crc_nncp.yaml 2026-03-09 19:19:31.981703 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/openstack/ 2026-03-09 19:19:31.981725 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/openstack/cr/ 2026-03-09 19:19:31.981747 | controller | cd+++++++++ ci-framework-data/artifacts/parameters/ 2026-03-09 19:19:31.981776 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/custom-params.yml 2026-03-09 19:19:31.981804 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/install-yamls-params.yml 2026-03-09 19:19:31.981827 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/openshift-login-params.yml 2026-03-09 19:19:31.981853 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/zuul-params.yml 2026-03-09 19:19:31.981875 | controller | cd+++++++++ ci-framework-data/artifacts/repositories/ 2026-03-09 19:19:31.981897 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean-antelope-testing.repo 2026-03-09 19:19:31.981925 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean.repo 2026-03-09 19:19:31.981948 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean.repo.md5 2026-03-09 19:19:31.981971 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/gating.repo 2026-03-09 19:19:31.981993 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-appstream.repo 2026-03-09 19:19:31.982033 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-baseos.repo 2026-03-09 19:19:31.982062 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-highavailability.repo 2026-03-09 19:19:31.982085 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-powertools.repo 2026-03-09 19:19:31.982108 | controller | cd+++++++++ ci-framework-data/artifacts/roles/ 2026-03-09 19:19:31.982131 | controller | cd+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/ 2026-03-09 19:19:31.982153 | controller | cd+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/ 2026-03-09 19:19:31.982175 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_all.yml 2026-03-09 19:19:31.982197 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_attach_default_interface.yml 2026-03-09 19:19:31.982218 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_attach_default_interface_cleanup.yml 2026-03-09 19:19:31.982241 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican.yml 2026-03-09 19:19:31.982263 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_cleanup.yml 2026-03-09 19:19:31.982284 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy.yml 2026-03-09 19:19:31.982305 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_cleanup.yml 2026-03-09 19:19:31.982327 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_prep.yml 2026-03-09 19:19:31.982348 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_validate.yml 2026-03-09 19:19:31.982369 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_kuttl.yml 2026-03-09 19:19:31.982391 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_kuttl_run.yml 2026-03-09 19:19:31.982413 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_prep.yml 2026-03-09 19:19:31.982436 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal.yml 2026-03-09 19:19:31.982458 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal_cleanup.yml 2026-03-09 19:19:31.982480 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal_prep.yml 2026-03-09 19:19:31.982502 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas.yml 2026-03-09 19:19:31.982524 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_baremetal_net_nad.yml 2026-03-09 19:19:31.982546 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_baremetal_net_nad_cleanup.yml 2026-03-09 19:19:31.982569 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_cleanup.yml 2026-03-09 19:19:31.982591 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_attach_network.yml 2026-03-09 19:19:31.982612 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_attach_network_cleanup.yml 2026-03-09 19:19:31.982639 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_baremetal_bridge.yml 2026-03-09 19:19:31.982661 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_baremetal_bridge_cleanup.yml 2026-03-09 19:19:31.982683 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_generate_nodes_yaml.yml 2026-03-09 19:19:31.982704 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_metallb.yml 2026-03-09 19:19:31.982726 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_metallb_cleanup.yml 2026-03-09 19:19:31.982747 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_network.yml 2026-03-09 19:19:31.982769 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_network_cleanup.yml 2026-03-09 19:19:31.982791 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_route_crc_and_crc_bmaas_networks.yml 2026-03-09 19:19:31.982813 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_route_crc_and_crc_bmaas_networks_cleanup.yml 2026-03-09 19:19:31.982834 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator.yml 2026-03-09 19:19:31.982856 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator_cleanup.yml 2026-03-09 19:19:31.982878 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator_wait.yml 2026-03-09 19:19:31.982901 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_virtual_bms.yml 2026-03-09 19:19:31.982922 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_virtual_bms_cleanup.yml 2026-03-09 19:19:31.982944 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph.yml 2026-03-09 19:19:31.982966 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph_cleanup.yml 2026-03-09 19:19:31.982988 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph_help.yml 2026-03-09 19:19:31.983010 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_certmanager.yml 2026-03-09 19:19:31.983057 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_certmanager_cleanup.yml 2026-03-09 19:19:31.983080 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cifmw_cleanup.yml 2026-03-09 19:19:31.983102 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cifmw_prepare.yml 2026-03-09 19:19:31.983123 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder.yml 2026-03-09 19:19:31.983145 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_cleanup.yml 2026-03-09 19:19:31.983165 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy.yml 2026-03-09 19:19:31.983187 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy_cleanup.yml 2026-03-09 19:19:31.983208 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy_prep.yml 2026-03-09 19:19:31.983229 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_kuttl.yml 2026-03-09 19:19:31.983250 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_kuttl_run.yml 2026-03-09 19:19:31.983271 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_prep.yml 2026-03-09 19:19:31.983302 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cleanup.yml 2026-03-09 19:19:31.983324 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc.yml 2026-03-09 19:19:31.983346 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_attach_default_interface.yml 2026-03-09 19:19:31.983368 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_attach_default_interface_cleanup.yml 2026-03-09 19:19:31.983390 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_bmo_cleanup.yml 2026-03-09 19:19:31.983412 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_bmo_setup.yml 2026-03-09 19:19:31.983433 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_cleanup.yml 2026-03-09 19:19:31.983454 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_scrub.yml 2026-03-09 19:19:31.983478 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage.yml 2026-03-09 19:19:31.983500 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_cleanup.yml 2026-03-09 19:19:31.983523 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_cleanup_with_retries.yml 2026-03-09 19:19:31.983545 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_release.yml 2026-03-09 19:19:31.983584 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_with_retries.yml 2026-03-09 19:19:31.983607 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_deploy_cleanup.yml 2026-03-09 19:19:31.983629 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate.yml 2026-03-09 19:19:31.983652 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_cleanup.yml 2026-03-09 19:19:31.983674 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy.yml 2026-03-09 19:19:31.983696 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy_cleanup.yml 2026-03-09 19:19:31.983721 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy_prep.yml 2026-03-09 19:19:31.983743 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_kuttl.yml 2026-03-09 19:19:31.983764 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_kuttl_run.yml 2026-03-09 19:19:31.983785 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_prep.yml 2026-03-09 19:19:31.983807 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy.yml 2026-03-09 19:19:31.983828 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy_cleanup.yml 2026-03-09 19:19:31.983849 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy_prep.yml 2026-03-09 19:19:31.983870 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_download_tools.yml 2026-03-09 19:19:31.983891 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_ansible_runner.yml 2026-03-09 19:19:31.983913 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_baremetal_compute.yml 2026-03-09 19:19:31.983934 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute.yml 2026-03-09 19:19:31.983960 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_bootc.yml 2026-03-09 19:19:31.983982 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_cleanup.yml 2026-03-09 19:19:31.984004 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_repos.yml 2026-03-09 19:19:31.984051 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_computes_bgp.yml 2026-03-09 19:19:31.984076 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy.yml 2026-03-09 19:19:31.984101 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_baremetal.yml 2026-03-09 19:19:31.984125 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_baremetal_prep.yml 2026-03-09 19:19:31.984147 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_cleanup.yml 2026-03-09 19:19:31.984168 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_generate_keys.yml 2026-03-09 19:19:31.984194 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_instance.yml 2026-03-09 19:19:31.984216 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker.yml 2026-03-09 19:19:31.984237 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker_cleanup.yml 2026-03-09 19:19:31.984258 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker_prep.yml 2026-03-09 19:19:31.984279 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_prep.yml 2026-03-09 19:19:31.984300 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_networker.yml 2026-03-09 19:19:31.984322 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_networker_cleanup.yml 2026-03-09 19:19:31.984343 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_nova_discover_hosts.yml 2026-03-09 19:19:31.984364 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_patch_ansible_runner_image.yml 2026-03-09 19:19:31.984386 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_register_dns.yml 2026-03-09 19:19:31.984407 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_wait_deploy.yml 2026-03-09 19:19:31.984429 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_wait_deploy_baremetal.yml 2026-03-09 19:19:31.984450 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance.yml 2026-03-09 19:19:31.984471 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_cleanup.yml 2026-03-09 19:19:31.984492 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy.yml 2026-03-09 19:19:31.984512 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy_cleanup.yml 2026-03-09 19:19:31.984533 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy_prep.yml 2026-03-09 19:19:31.984554 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_kuttl.yml 2026-03-09 19:19:31.984575 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_kuttl_run.yml 2026-03-09 19:19:31.984596 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_prep.yml 2026-03-09 19:19:31.984623 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat.yml 2026-03-09 19:19:31.984645 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_cleanup.yml 2026-03-09 19:19:31.984666 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy.yml 2026-03-09 19:19:31.984688 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy_cleanup.yml 2026-03-09 19:19:31.984709 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy_prep.yml 2026-03-09 19:19:31.984729 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl.yml 2026-03-09 19:19:31.984750 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl_crc.yml 2026-03-09 19:19:31.984770 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl_run.yml 2026-03-09 19:19:31.984791 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_prep.yml 2026-03-09 19:19:31.984812 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_help.yml 2026-03-09 19:19:31.984833 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon.yml 2026-03-09 19:19:31.984854 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_cleanup.yml 2026-03-09 19:19:31.984875 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy.yml 2026-03-09 19:19:31.984896 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy_cleanup.yml 2026-03-09 19:19:31.984917 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy_prep.yml 2026-03-09 19:19:31.984938 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_kuttl.yml 2026-03-09 19:19:31.984959 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_kuttl_run.yml 2026-03-09 19:19:31.984980 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_prep.yml 2026-03-09 19:19:31.985001 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra.yml 2026-03-09 19:19:31.985051 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_cleanup.yml 2026-03-09 19:19:31.985076 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_kuttl.yml 2026-03-09 19:19:31.985098 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_kuttl_run.yml 2026-03-09 19:19:31.985120 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_prep.yml 2026-03-09 19:19:31.985141 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_input.yml 2026-03-09 19:19:31.985162 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_input_cleanup.yml 2026-03-09 19:19:31.985184 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab.yml 2026-03-09 19:19:31.985205 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_cleanup.yml 2026-03-09 19:19:31.985227 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_nat64_router.yml 2026-03-09 19:19:31.985249 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_nat64_router_cleanup.yml 2026-03-09 19:19:31.985277 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_network.yml 2026-03-09 19:19:31.985299 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_network_cleanup.yml 2026-03-09 19:19:31.985321 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_sno.yml 2026-03-09 19:19:31.985343 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_sno_cleanup.yml 2026-03-09 19:19:31.985365 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic.yml 2026-03-09 19:19:31.985387 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_cleanup.yml 2026-03-09 19:19:31.985409 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy.yml 2026-03-09 19:19:31.985430 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy_cleanup.yml 2026-03-09 19:19:31.985452 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy_prep.yml 2026-03-09 19:19:31.985474 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl.yml 2026-03-09 19:19:31.985495 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl_crc.yml 2026-03-09 19:19:31.985517 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl_run.yml 2026-03-09 19:19:31.985538 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_prep.yml 2026-03-09 19:19:31.985560 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone.yml 2026-03-09 19:19:31.985582 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_cleanup.yml 2026-03-09 19:19:31.985604 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy.yml 2026-03-09 19:19:31.985626 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy_cleanup.yml 2026-03-09 19:19:31.985647 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy_prep.yml 2026-03-09 19:19:31.985669 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_kuttl.yml 2026-03-09 19:19:31.985690 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_kuttl_run.yml 2026-03-09 19:19:31.985711 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_prep.yml 2026-03-09 19:19:31.985732 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_common_cleanup.yml 2026-03-09 19:19:31.985754 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_common_prep.yml 2026-03-09 19:19:31.985775 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_db_cleanup.yml 2026-03-09 19:19:31.985796 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_db_prep.yml 2026-03-09 19:19:31.985818 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki.yml 2026-03-09 19:19:31.985839 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_cleanup.yml 2026-03-09 19:19:31.985860 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_deploy.yml 2026-03-09 19:19:31.985881 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_deploy_cleanup.yml 2026-03-09 19:19:31.985903 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_lvms.yml 2026-03-09 19:19:31.985929 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila.yml 2026-03-09 19:19:31.985951 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_cleanup.yml 2026-03-09 19:19:31.985972 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy.yml 2026-03-09 19:19:31.985993 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy_cleanup.yml 2026-03-09 19:19:31.986032 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy_prep.yml 2026-03-09 19:19:31.986061 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_kuttl.yml 2026-03-09 19:19:31.986083 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_kuttl_run.yml 2026-03-09 19:19:31.986105 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_prep.yml 2026-03-09 19:19:31.986127 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb.yml 2026-03-09 19:19:31.986149 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_chainsaw.yml 2026-03-09 19:19:31.986170 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_chainsaw_run.yml 2026-03-09 19:19:31.986192 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_cleanup.yml 2026-03-09 19:19:31.986214 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy.yml 2026-03-09 19:19:31.986235 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy_cleanup.yml 2026-03-09 19:19:31.986256 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy_prep.yml 2026-03-09 19:19:31.986278 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_kuttl.yml 2026-03-09 19:19:31.986299 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_kuttl_run.yml 2026-03-09 19:19:31.986321 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy.yml 2026-03-09 19:19:31.986342 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy_cleanup.yml 2026-03-09 19:19:31.986363 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy_prep.yml 2026-03-09 19:19:31.986383 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb.yml 2026-03-09 19:19:31.986404 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_cleanup.yml 2026-03-09 19:19:31.986424 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_config.yml 2026-03-09 19:19:31.986445 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_config_cleanup.yml 2026-03-09 19:19:31.986466 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry.yml 2026-03-09 19:19:31.986486 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_ca.yml 2026-03-09 19:19:31.986507 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_cleanup.yml 2026-03-09 19:19:31.986527 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_secure.yml 2026-03-09 19:19:31.986548 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_namespace.yml 2026-03-09 19:19:31.986573 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_namespace_cleanup.yml 2026-03-09 19:19:31.986595 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netattach.yml 2026-03-09 19:19:31.986615 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netattach_cleanup.yml 2026-03-09 19:19:31.986636 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy.yml 2026-03-09 19:19:31.986657 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy_cleanup.yml 2026-03-09 19:19:31.986679 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy_prep.yml 2026-03-09 19:19:31.986700 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv.yml 2026-03-09 19:19:31.986722 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_cleanup.yml 2026-03-09 19:19:31.986744 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_deploy.yml 2026-03-09 19:19:31.986766 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_deploy_cleanup.yml 2026-03-09 19:19:31.986788 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_network_isolation_bridge.yml 2026-03-09 19:19:31.986810 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_network_isolation_bridge_cleanup.yml 2026-03-09 19:19:31.986832 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron.yml 2026-03-09 19:19:31.986854 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_cleanup.yml 2026-03-09 19:19:31.986875 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy.yml 2026-03-09 19:19:31.986896 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy_cleanup.yml 2026-03-09 19:19:31.986917 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy_prep.yml 2026-03-09 19:19:31.986938 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_kuttl.yml 2026-03-09 19:19:31.986960 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_kuttl_run.yml 2026-03-09 19:19:31.986981 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_prep.yml 2026-03-09 19:19:31.987002 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nfs.yml 2026-03-09 19:19:31.987048 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nfs_cleanup.yml 2026-03-09 19:19:31.987073 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nmstate.yml 2026-03-09 19:19:31.987096 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nncp.yml 2026-03-09 19:19:31.987118 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nncp_cleanup.yml 2026-03-09 19:19:31.987140 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova.yml 2026-03-09 19:19:31.987162 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_cleanup.yml 2026-03-09 19:19:31.987183 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy.yml 2026-03-09 19:19:31.987205 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy_cleanup.yml 2026-03-09 19:19:31.987226 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy_prep.yml 2026-03-09 19:19:31.987252 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_prep.yml 2026-03-09 19:19:31.987274 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia.yml 2026-03-09 19:19:31.987295 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_cleanup.yml 2026-03-09 19:19:31.987316 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy.yml 2026-03-09 19:19:31.987338 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy_cleanup.yml 2026-03-09 19:19:31.987359 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy_prep.yml 2026-03-09 19:19:31.987381 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_kuttl.yml 2026-03-09 19:19:31.987402 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_kuttl_run.yml 2026-03-09 19:19:31.987423 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_prep.yml 2026-03-09 19:19:31.987445 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack.yml 2026-03-09 19:19:31.987465 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_cleanup.yml 2026-03-09 19:19:31.987487 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_crds.yml 2026-03-09 19:19:31.987508 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_crds_cleanup.yml 2026-03-09 19:19:31.987529 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy.yml 2026-03-09 19:19:31.987562 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy_cleanup.yml 2026-03-09 19:19:31.987585 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy_prep.yml 2026-03-09 19:19:31.987607 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_init.yml 2026-03-09 19:19:31.987629 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl.yml 2026-03-09 19:19:31.987649 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_cleanup.yml 2026-03-09 19:19:31.987670 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_prep.yml 2026-03-09 19:19:31.987691 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_run.yml 2026-03-09 19:19:31.987711 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_patch_version.yml 2026-03-09 19:19:31.987732 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_prep.yml 2026-03-09 19:19:31.987753 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_repo.yml 2026-03-09 19:19:31.987774 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_update_run.yml 2026-03-09 19:19:31.987795 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_wait.yml 2026-03-09 19:19:31.987816 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_wait_deploy.yml 2026-03-09 19:19:31.987837 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_operator_namespace.yml 2026-03-09 19:19:31.987858 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn.yml 2026-03-09 19:19:31.987884 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_cleanup.yml 2026-03-09 19:19:31.987906 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy.yml 2026-03-09 19:19:31.987927 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy_cleanup.yml 2026-03-09 19:19:31.987949 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy_prep.yml 2026-03-09 19:19:31.987970 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_kuttl.yml 2026-03-09 19:19:31.987990 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_kuttl_run.yml 2026-03-09 19:19:31.988011 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_prep.yml 2026-03-09 19:19:31.988058 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement.yml 2026-03-09 19:19:31.988084 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_cleanup.yml 2026-03-09 19:19:31.988107 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy.yml 2026-03-09 19:19:31.988128 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy_cleanup.yml 2026-03-09 19:19:31.988149 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy_prep.yml 2026-03-09 19:19:31.988170 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_kuttl.yml 2026-03-09 19:19:31.988191 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_kuttl_run.yml 2026-03-09 19:19:31.988212 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_prep.yml 2026-03-09 19:19:31.988233 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq.yml 2026-03-09 19:19:31.988255 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_cleanup.yml 2026-03-09 19:19:31.988277 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy.yml 2026-03-09 19:19:31.988299 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy_cleanup.yml 2026-03-09 19:19:31.988321 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy_prep.yml 2026-03-09 19:19:31.988343 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_prep.yml 2026-03-09 19:19:31.988364 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy.yml 2026-03-09 19:19:31.988386 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy_cleanup.yml 2026-03-09 19:19:31.988407 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy_prep.yml 2026-03-09 19:19:31.988428 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook.yml 2026-03-09 19:19:31.988449 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_cleanup.yml 2026-03-09 19:19:31.988470 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_crc_disk.yml 2026-03-09 19:19:31.988490 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_deploy.yml 2026-03-09 19:19:31.988510 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_deploy_prep.yml 2026-03-09 19:19:31.988531 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_prep.yml 2026-03-09 19:19:31.988557 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_set_slower_etcd_profile.yml 2026-03-09 19:19:31.988580 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone.yml 2026-03-09 19:19:31.988602 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_cleanup.yml 2026-03-09 19:19:31.988624 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_deploy.yml 2026-03-09 19:19:31.988646 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_revert.yml 2026-03-09 19:19:31.988667 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_snapshot.yml 2026-03-09 19:19:31.988688 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_sync.yml 2026-03-09 19:19:31.988710 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift.yml 2026-03-09 19:19:31.988731 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_cleanup.yml 2026-03-09 19:19:31.988752 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy.yml 2026-03-09 19:19:31.988774 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy_cleanup.yml 2026-03-09 19:19:31.988796 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy_prep.yml 2026-03-09 19:19:31.988817 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_kuttl.yml 2026-03-09 19:19:31.988838 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_kuttl_run.yml 2026-03-09 19:19:31.988860 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_prep.yml 2026-03-09 19:19:31.988881 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry.yml 2026-03-09 19:19:31.988902 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_cleanup.yml 2026-03-09 19:19:31.988924 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy.yml 2026-03-09 19:19:31.988945 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy_cleanup.yml 2026-03-09 19:19:31.988966 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy_prep.yml 2026-03-09 19:19:31.988987 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_kuttl.yml 2026-03-09 19:19:31.989008 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_kuttl_run.yml 2026-03-09 19:19:31.989054 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_prep.yml 2026-03-09 19:19:31.989077 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_tripleo_deploy.yml 2026-03-09 19:19:31.989098 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_update_services.yml 2026-03-09 19:19:31.989120 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_update_system.yml 2026-03-09 19:19:31.989141 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_validate_marketplace.yml 2026-03-09 19:19:31.989163 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_wait.yml 2026-03-09 19:19:31.989187 | controller | cd+++++++++ ci-framework-data/artifacts/yum_repos/ 2026-03-09 19:19:31.989216 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean-antelope-testing.repo 2026-03-09 19:19:31.989240 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean.repo 2026-03-09 19:19:31.989263 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean.repo.md5 2026-03-09 19:19:31.989286 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/gating.repo 2026-03-09 19:19:31.989308 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-appstream.repo 2026-03-09 19:19:31.989420 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-baseos.repo 2026-03-09 19:19:31.989453 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-highavailability.repo 2026-03-09 19:19:31.989479 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-powertools.repo 2026-03-09 19:19:31.989503 | controller | cd+++++++++ ci-framework-data/logs/ 2026-03-09 19:19:31.989527 | controller | >f+++++++++ ci-framework-data/logs/ansible.log.gz 2026-03-09 19:19:31.989550 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_copy_logs_from_crc.log 2026-03-09 19:19:31.989572 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_prepare_root_ssh.log 2026-03-09 19:19:31.989595 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_run_download.log 2026-03-09 19:19:31.989617 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_run_openstack_must_gather.log 2026-03-09 19:19:31.989639 | controller | >f+++++++++ ci-framework-data/logs/ci_script_001_run_hook_without_retry.log 2026-03-09 19:19:31.989661 | controller | >f+++++++++ ci-framework-data/logs/ci_script_002_fetch_openshift.log 2026-03-09 19:19:31.989683 | controller | >f+++++++++ ci-framework-data/logs/ci_script_003_run_openstack.log 2026-03-09 19:19:31.989704 | controller | >f+++++++++ ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log 2026-03-09 19:19:31.989726 | controller | >f+++++++++ ci-framework-data/logs/ci_script_005_stop_watcher.log 2026-03-09 19:19:31.989747 | controller | >f+++++++++ ci-framework-data/logs/ci_script_006_install_watcher_operator_in.log 2026-03-09 19:19:31.989769 | controller | >f+++++++++ ci-framework-data/logs/post_install_operators_kuttl_from_operator_deploy_watcher_operator.log 2026-03-09 19:19:31.989790 | controller | >f+++++++++ ci-framework-data/logs/pre_infra_download_needed_tools.log 2026-03-09 19:19:31.989812 | controller | cd+++++++++ ci-framework-data/logs/2026-03-09_19-10/ 2026-03-09 19:19:31.989833 | controller | >f+++++++++ ci-framework-data/logs/2026-03-09_19-10/ansible.log.gz 2026-03-09 19:19:31.989855 | controller | cd+++++++++ ci-framework-data/logs/crc/ 2026-03-09 19:19:31.989876 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/ 2026-03-09 19:19:31.989898 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/ 2026-03-09 19:19:31.989920 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-fzzrb_ce95d0cc-b615-4635-ba32-5a652793187b/ 2026-03-09 19:19:31.989943 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-fzzrb_ce95d0cc-b615-4635-ba32-5a652793187b/cert-manager-operator/ 2026-03-09 19:19:31.989965 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-fzzrb_ce95d0cc-b615-4635-ba32-5a652793187b/cert-manager-operator/0.log 2026-03-09 19:19:31.989986 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-zmpfr_c9cd2b98-2171-4c11-abb5-a0e3db0a69d5/ 2026-03-09 19:19:31.990009 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-zmpfr_c9cd2b98-2171-4c11-abb5-a0e3db0a69d5/cert-manager-controller/ 2026-03-09 19:19:31.990065 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-zmpfr_c9cd2b98-2171-4c11-abb5-a0e3db0a69d5/cert-manager-controller/0.log 2026-03-09 19:19:31.990093 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-2t9gn_2840cece-7d09-420e-8c47-85417d8032a9/ 2026-03-09 19:19:31.990115 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-2t9gn_2840cece-7d09-420e-8c47-85417d8032a9/cert-manager-cainjector/ 2026-03-09 19:19:31.990137 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-2t9gn_2840cece-7d09-420e-8c47-85417d8032a9/cert-manager-cainjector/0.log 2026-03-09 19:19:31.990167 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-8k2s6_56627852-72af-4929-a17f-29e6675fdbfc/ 2026-03-09 19:19:31.990188 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-8k2s6_56627852-72af-4929-a17f-29e6675fdbfc/cert-manager-webhook/ 2026-03-09 19:19:31.990210 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-8k2s6_56627852-72af-4929-a17f-29e6675fdbfc/cert-manager-webhook/0.log 2026-03-09 19:19:31.990231 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/ 2026-03-09 19:19:31.990253 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/csi-provisioner/ 2026-03-09 19:19:31.990275 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/csi-provisioner/0.log 2026-03-09 19:19:31.990297 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/hostpath-provisioner/ 2026-03-09 19:19:31.990319 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/hostpath-provisioner/0.log 2026-03-09 19:19:31.990341 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/liveness-probe/ 2026-03-09 19:19:31.990363 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/liveness-probe/0.log 2026-03-09 19:19:31.990385 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/node-driver-registrar/ 2026-03-09 19:19:31.990407 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-vdrsg_75d58d1e-e673-4305-9d09-2cfd323769fd/node-driver-registrar/0.log 2026-03-09 19:19:31.990428 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-86ddb6bd46-jfl8k_b43e0c69-2c5a-4562-8ab9-d4a8d6e5404b/ 2026-03-09 19:19:31.990450 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-86ddb6bd46-jfl8k_b43e0c69-2c5a-4562-8ab9-d4a8d6e5404b/controller/ 2026-03-09 19:19:31.990471 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-86ddb6bd46-jfl8k_b43e0c69-2c5a-4562-8ab9-d4a8d6e5404b/controller/0.log.gz 2026-03-09 19:19:31.990493 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-86ddb6bd46-jfl8k_b43e0c69-2c5a-4562-8ab9-d4a8d6e5404b/kube-rbac-proxy/ 2026-03-09 19:19:31.990523 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-86ddb6bd46-jfl8k_b43e0c69-2c5a-4562-8ab9-d4a8d6e5404b/kube-rbac-proxy/0.log 2026-03-09 19:19:31.990545 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/ 2026-03-09 19:19:31.990567 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/3fd30038278bb0f87de5e05c0b6af7985e58c88786d95bc22d77dd8ab721051d.log 2026-03-09 19:19:31.990589 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/controller/ 2026-03-09 19:19:31.990611 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/controller/0.log 2026-03-09 19:19:31.990632 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/cp-frr-files/ 2026-03-09 19:19:31.990653 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/cp-frr-files/0.log 2026-03-09 19:19:31.990674 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/cp-metrics/ 2026-03-09 19:19:31.990696 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/cp-metrics/0.log 2026-03-09 19:19:31.990717 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/cp-reloader/ 2026-03-09 19:19:31.990738 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/cp-reloader/0.log 2026-03-09 19:19:31.990759 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/frr-metrics/ 2026-03-09 19:19:31.990780 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/frr-metrics/0.log 2026-03-09 19:19:31.990801 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/frr/ 2026-03-09 19:19:31.990823 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/frr/0.log.gz 2026-03-09 19:19:31.990844 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/kube-rbac-proxy-frr/ 2026-03-09 19:19:31.990866 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/kube-rbac-proxy-frr/0.log 2026-03-09 19:19:31.990888 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/kube-rbac-proxy/ 2026-03-09 19:19:31.990909 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/kube-rbac-proxy/0.log 2026-03-09 19:19:31.990930 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/reloader/ 2026-03-09 19:19:31.990951 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-pxc5m_ea8f9c80-04cb-455e-a2fc-2ed5b028a79c/reloader/0.log 2026-03-09 19:19:31.990972 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-7f989f654f-7q47l_64621269-3b51-4cc2-89c8-0fd5ad067fd7/ 2026-03-09 19:19:31.990998 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-7f989f654f-7q47l_64621269-3b51-4cc2-89c8-0fd5ad067fd7/frr-k8s-webhook-server/ 2026-03-09 19:19:31.991043 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-7f989f654f-7q47l_64621269-3b51-4cc2-89c8-0fd5ad067fd7/frr-k8s-webhook-server/0.log 2026-03-09 19:19:31.991075 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-858bc4f469-wp8gj_ece940b4-1c75-4a27-af76-1d0987599334/ 2026-03-09 19:19:31.991097 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-858bc4f469-wp8gj_ece940b4-1c75-4a27-af76-1d0987599334/manager/ 2026-03-09 19:19:31.991203 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-858bc4f469-wp8gj_ece940b4-1c75-4a27-af76-1d0987599334/manager/0.log 2026-03-09 19:19:31.991235 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-5f89859c4b-c6xkg_57e177f6-8afa-42f4-ac0c-2b43f01cf06a/ 2026-03-09 19:19:31.991260 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-5f89859c4b-c6xkg_57e177f6-8afa-42f4-ac0c-2b43f01cf06a/webhook-server/ 2026-03-09 19:19:31.991284 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-5f89859c4b-c6xkg_57e177f6-8afa-42f4-ac0c-2b43f01cf06a/webhook-server/0.log 2026-03-09 19:19:31.991307 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-5sdkw_cd899ccb-4a21-4e1f-93a3-39451435e6f8/ 2026-03-09 19:19:31.991330 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-5sdkw_cd899ccb-4a21-4e1f-93a3-39451435e6f8/kube-rbac-proxy/ 2026-03-09 19:19:31.991353 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-5sdkw_cd899ccb-4a21-4e1f-93a3-39451435e6f8/kube-rbac-proxy/0.log 2026-03-09 19:19:31.991375 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-5sdkw_cd899ccb-4a21-4e1f-93a3-39451435e6f8/speaker/ 2026-03-09 19:19:31.991397 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-5sdkw_cd899ccb-4a21-4e1f-93a3-39451435e6f8/speaker/0.log.gz 2026-03-09 19:19:31.991426 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-vlwv5_24aa1fc6-da2a-400c-8bfe-022af0ee3707/ 2026-03-09 19:19:31.991449 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-vlwv5_24aa1fc6-da2a-400c-8bfe-022af0ee3707/openshift-apiserver-operator/ 2026-03-09 19:19:31.991471 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-vlwv5_24aa1fc6-da2a-400c-8bfe-022af0ee3707/openshift-apiserver-operator/0.log 2026-03-09 19:19:31.991493 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-m6q6r_ea3fa689-2665-423f-b717-f2e279be3831/ 2026-03-09 19:19:31.991514 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-m6q6r_ea3fa689-2665-423f-b717-f2e279be3831/fix-audit-permissions/ 2026-03-09 19:19:31.991535 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-m6q6r_ea3fa689-2665-423f-b717-f2e279be3831/fix-audit-permissions/0.log 2026-03-09 19:19:31.991577 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-m6q6r_ea3fa689-2665-423f-b717-f2e279be3831/openshift-apiserver-check-endpoints/ 2026-03-09 19:19:31.991602 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-m6q6r_ea3fa689-2665-423f-b717-f2e279be3831/openshift-apiserver-check-endpoints/0.log 2026-03-09 19:19:31.991627 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-m6q6r_ea3fa689-2665-423f-b717-f2e279be3831/openshift-apiserver/ 2026-03-09 19:19:31.991650 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-m6q6r_ea3fa689-2665-423f-b717-f2e279be3831/openshift-apiserver/0.log 2026-03-09 19:19:31.991672 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-jrr9g_cebe05d8-86f1-4280-9ae0-8065f9c38759/ 2026-03-09 19:19:31.991694 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-jrr9g_cebe05d8-86f1-4280-9ae0-8065f9c38759/authentication-operator/ 2026-03-09 19:19:31.991715 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-jrr9g_cebe05d8-86f1-4280-9ae0-8065f9c38759/authentication-operator/0.log 2026-03-09 19:19:31.991737 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-68945756f9-mpsgc_d3ee9323-1645-46d9-a4e7-e721976401e0/ 2026-03-09 19:19:31.991759 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-68945756f9-mpsgc_d3ee9323-1645-46d9-a4e7-e721976401e0/oauth-openshift/ 2026-03-09 19:19:31.991781 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-68945756f9-mpsgc_d3ee9323-1645-46d9-a4e7-e721976401e0/oauth-openshift/0.log 2026-03-09 19:19:31.991802 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-nprxk_1b7ef6fd-c836-460d-bac0-ac2135ad77a2/ 2026-03-09 19:19:31.991824 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-nprxk_1b7ef6fd-c836-460d-bac0-ac2135ad77a2/kube-rbac-proxy/ 2026-03-09 19:19:31.991847 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-nprxk_1b7ef6fd-c836-460d-bac0-ac2135ad77a2/kube-rbac-proxy/0.log 2026-03-09 19:19:31.991869 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-nprxk_1b7ef6fd-c836-460d-bac0-ac2135ad77a2/machine-approver-controller/ 2026-03-09 19:19:31.991891 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-nprxk_1b7ef6fd-c836-460d-bac0-ac2135ad77a2/machine-approver-controller/0.log 2026-03-09 19:19:31.991912 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-gbjt5_a663703c-95db-4871-b31c-00951488935d/ 2026-03-09 19:19:31.991935 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-gbjt5_a663703c-95db-4871-b31c-00951488935d/574ad56816421dbef1b3c2e14da70c046c125fd229ffdd3d8466e6b34e732f21.log 2026-03-09 19:19:31.991958 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-gbjt5_a663703c-95db-4871-b31c-00951488935d/cluster-samples-operator-watch/ 2026-03-09 19:19:31.991986 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-gbjt5_a663703c-95db-4871-b31c-00951488935d/cluster-samples-operator-watch/0.log 2026-03-09 19:19:31.992009 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-gbjt5_a663703c-95db-4871-b31c-00951488935d/cluster-samples-operator/ 2026-03-09 19:19:31.992057 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-gbjt5_a663703c-95db-4871-b31c-00951488935d/cluster-samples-operator/0.log 2026-03-09 19:19:31.992081 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-gbjt5_a663703c-95db-4871-b31c-00951488935d/cluster-samples-operator/1.log 2026-03-09 19:19:31.992103 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-qnxgv_489cb5f6-bb44-4678-80cb-e399f23658bc/ 2026-03-09 19:19:31.992125 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-qnxgv_489cb5f6-bb44-4678-80cb-e399f23658bc/cluster-version-operator/ 2026-03-09 19:19:31.992147 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-qnxgv_489cb5f6-bb44-4678-80cb-e399f23658bc/cluster-version-operator/0.log.gz 2026-03-09 19:19:31.992169 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-l97hl_8886c330-bce2-4801-be16-59eeddddaf6f/ 2026-03-09 19:19:31.992191 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-l97hl_8886c330-bce2-4801-be16-59eeddddaf6f/openshift-api/ 2026-03-09 19:19:31.992213 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-l97hl_8886c330-bce2-4801-be16-59eeddddaf6f/openshift-api/0.log 2026-03-09 19:19:31.992235 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-l97hl_8886c330-bce2-4801-be16-59eeddddaf6f/openshift-config-operator/ 2026-03-09 19:19:31.992257 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-l97hl_8886c330-bce2-4801-be16-59eeddddaf6f/openshift-config-operator/0.log 2026-03-09 19:19:31.992279 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-gg4ds_934a74a7-234e-44f1-bc6e-a13661836b6b/ 2026-03-09 19:19:31.992301 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-gg4ds_934a74a7-234e-44f1-bc6e-a13661836b6b/console-operator/ 2026-03-09 19:19:31.992323 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-gg4ds_934a74a7-234e-44f1-bc6e-a13661836b6b/console-operator/0.log 2026-03-09 19:19:31.992344 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-859b85d6d-tltz7_0a0c8b8c-30fa-41b0-8598-b8cfc6b1455f/ 2026-03-09 19:19:31.992366 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-859b85d6d-tltz7_0a0c8b8c-30fa-41b0-8598-b8cfc6b1455f/console/ 2026-03-09 19:19:31.992387 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-859b85d6d-tltz7_0a0c8b8c-30fa-41b0-8598-b8cfc6b1455f/console/0.log 2026-03-09 19:19:31.992414 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-295wb_f078c2bb-b4ba-42a0-a66c-705c19866fec/ 2026-03-09 19:19:31.992436 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-295wb_f078c2bb-b4ba-42a0-a66c-705c19866fec/download-server/ 2026-03-09 19:19:31.992458 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-295wb_f078c2bb-b4ba-42a0-a66c-705c19866fec/download-server/0.log 2026-03-09 19:19:31.992480 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-9pcrm_a423d95a-7bd6-483e-ba23-28e8f1a3ec92/ 2026-03-09 19:19:31.992502 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-9pcrm_a423d95a-7bd6-483e-ba23-28e8f1a3ec92/openshift-controller-manager-operator/ 2026-03-09 19:19:31.992524 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-9pcrm_a423d95a-7bd6-483e-ba23-28e8f1a3ec92/openshift-controller-manager-operator/0.log 2026-03-09 19:19:31.992546 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-548b5494f8-8sdc6_605e5144-5967-4913-a269-a024956c1911/ 2026-03-09 19:19:31.992568 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-548b5494f8-8sdc6_605e5144-5967-4913-a269-a024956c1911/controller-manager/ 2026-03-09 19:19:31.992590 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-548b5494f8-8sdc6_605e5144-5967-4913-a269-a024956c1911/controller-manager/0.log 2026-03-09 19:19:31.992612 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-znqzp_87962440-47ce-4659-a2a7-f00110cc3bd5/ 2026-03-09 19:19:31.992633 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-znqzp_87962440-47ce-4659-a2a7-f00110cc3bd5/dns-operator/ 2026-03-09 19:19:31.992655 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-znqzp_87962440-47ce-4659-a2a7-f00110cc3bd5/dns-operator/0.log 2026-03-09 19:19:31.992676 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-znqzp_87962440-47ce-4659-a2a7-f00110cc3bd5/kube-rbac-proxy/ 2026-03-09 19:19:31.992698 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-znqzp_87962440-47ce-4659-a2a7-f00110cc3bd5/kube-rbac-proxy/0.log 2026-03-09 19:19:31.992719 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-c257s_8e5b560f-cc32-4a1a-8632-383befaabb5a/ 2026-03-09 19:19:31.992740 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-c257s_8e5b560f-cc32-4a1a-8632-383befaabb5a/dns/ 2026-03-09 19:19:31.992762 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-c257s_8e5b560f-cc32-4a1a-8632-383befaabb5a/dns/0.log 2026-03-09 19:19:31.992783 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-c257s_8e5b560f-cc32-4a1a-8632-383befaabb5a/kube-rbac-proxy/ 2026-03-09 19:19:31.992804 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-c257s_8e5b560f-cc32-4a1a-8632-383befaabb5a/kube-rbac-proxy/0.log 2026-03-09 19:19:31.992830 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-n9tvt_b53a5b8b-3dab-4300-8b7b-c3df20eab3b7/ 2026-03-09 19:19:31.992852 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-n9tvt_b53a5b8b-3dab-4300-8b7b-c3df20eab3b7/dns-node-resolver/ 2026-03-09 19:19:31.992874 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-n9tvt_b53a5b8b-3dab-4300-8b7b-c3df20eab3b7/dns-node-resolver/0.log 2026-03-09 19:19:31.992895 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-sh5rp_f080ed0b-402b-4ed1-89cd-5ee1af2b9735/ 2026-03-09 19:19:31.992917 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-sh5rp_f080ed0b-402b-4ed1-89cd-5ee1af2b9735/etcd-operator/ 2026-03-09 19:19:31.992938 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-sh5rp_f080ed0b-402b-4ed1-89cd-5ee1af2b9735/etcd-operator/0.log 2026-03-09 19:19:31.992960 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/ 2026-03-09 19:19:31.992981 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-ensure-env-vars/ 2026-03-09 19:19:31.993003 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-ensure-env-vars/0.log 2026-03-09 19:19:31.993049 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-metrics/ 2026-03-09 19:19:31.993073 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-metrics/0.log 2026-03-09 19:19:31.993098 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-readyz/ 2026-03-09 19:19:31.993122 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-readyz/0.log 2026-03-09 19:19:31.993143 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-resources-copy/ 2026-03-09 19:19:31.993165 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-resources-copy/0.log 2026-03-09 19:19:31.993186 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-rev/ 2026-03-09 19:19:31.993208 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-rev/0.log 2026-03-09 19:19:31.993230 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd/ 2026-03-09 19:19:31.993252 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd/0.log 2026-03-09 19:19:31.993274 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcdctl/ 2026-03-09 19:19:31.993295 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcdctl/0.log 2026-03-09 19:19:31.993317 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/setup/ 2026-03-09 19:19:31.993338 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/setup/0.log 2026-03-09 19:19:31.993365 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-66c4m_a3e149e2-c719-4025-888c-3134dd07b7c4/ 2026-03-09 19:19:31.993388 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-66c4m_a3e149e2-c719-4025-888c-3134dd07b7c4/cluster-image-registry-operator/ 2026-03-09 19:19:31.993410 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-66c4m_a3e149e2-c719-4025-888c-3134dd07b7c4/cluster-image-registry-operator/0.log 2026-03-09 19:19:31.993432 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-sr524_8a08d45c-ab67-496b-80e7-9f630d75e6cf/ 2026-03-09 19:19:31.993455 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-sr524_8a08d45c-ab67-496b-80e7-9f630d75e6cf/registry/ 2026-03-09 19:19:31.993477 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-sr524_8a08d45c-ab67-496b-80e7-9f630d75e6cf/registry/0.log 2026-03-09 19:19:31.993499 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-mfdmq_a085b570-506c-4b51-b0d1-4b9832e71c0f/ 2026-03-09 19:19:31.993521 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-mfdmq_a085b570-506c-4b51-b0d1-4b9832e71c0f/node-ca/ 2026-03-09 19:19:31.993544 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-mfdmq_a085b570-506c-4b51-b0d1-4b9832e71c0f/node-ca/0.log 2026-03-09 19:19:31.993567 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551390-ngv5v_ae07b7e5-6cfa-40bb-9c95-be56354dd2fd/ 2026-03-09 19:19:31.993589 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551390-ngv5v_ae07b7e5-6cfa-40bb-9c95-be56354dd2fd/oc/ 2026-03-09 19:19:31.993611 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551390-ngv5v_ae07b7e5-6cfa-40bb-9c95-be56354dd2fd/oc/0.log 2026-03-09 19:19:31.993632 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551392-f2pbf_fe2f8900-837d-4b97-81c8-1ebb0f5a49bd/ 2026-03-09 19:19:31.993653 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551392-f2pbf_fe2f8900-837d-4b97-81c8-1ebb0f5a49bd/oc/ 2026-03-09 19:19:31.993675 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551392-f2pbf_fe2f8900-837d-4b97-81c8-1ebb0f5a49bd/oc/0.log 2026-03-09 19:19:31.993696 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551394-4fbs7_dfd10209-072d-4352-9d8b-72b620a1d174/ 2026-03-09 19:19:31.993717 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551394-4fbs7_dfd10209-072d-4352-9d8b-72b620a1d174/oc/ 2026-03-09 19:19:31.993738 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29551394-4fbs7_dfd10209-072d-4352-9d8b-72b620a1d174/oc/0.log 2026-03-09 19:19:31.993759 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-djphk_2e487bc2-9b7d-4845-a026-b27c82e6257a/ 2026-03-09 19:19:31.993780 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-djphk_2e487bc2-9b7d-4845-a026-b27c82e6257a/serve-healthcheck-canary/ 2026-03-09 19:19:31.993806 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-djphk_2e487bc2-9b7d-4845-a026-b27c82e6257a/serve-healthcheck-canary/0.log 2026-03-09 19:19:31.993827 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-7fdbf_b6b5dbe9-77c4-4cd4-b639-ade5dff8134c/ 2026-03-09 19:19:31.993849 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-7fdbf_b6b5dbe9-77c4-4cd4-b639-ade5dff8134c/ingress-operator/ 2026-03-09 19:19:31.993870 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-7fdbf_b6b5dbe9-77c4-4cd4-b639-ade5dff8134c/ingress-operator/0.log 2026-03-09 19:19:31.993891 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-7fdbf_b6b5dbe9-77c4-4cd4-b639-ade5dff8134c/kube-rbac-proxy/ 2026-03-09 19:19:31.993912 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-7fdbf_b6b5dbe9-77c4-4cd4-b639-ade5dff8134c/kube-rbac-proxy/0.log 2026-03-09 19:19:31.993933 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-4ntmx_a28f17d7-69dc-4014-a347-a26f55d55ace/ 2026-03-09 19:19:31.993954 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-4ntmx_a28f17d7-69dc-4014-a347-a26f55d55ace/router/ 2026-03-09 19:19:31.993975 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-4ntmx_a28f17d7-69dc-4014-a347-a26f55d55ace/router/0.log 2026-03-09 19:19:31.993996 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-xxsvf_e3f4a1f9-c144-4ccb-ba8a-1a7861a5b665/ 2026-03-09 19:19:31.994035 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-xxsvf_e3f4a1f9-c144-4ccb-ba8a-1a7861a5b665/kube-apiserver-operator/ 2026-03-09 19:19:31.994063 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-xxsvf_e3f4a1f9-c144-4ccb-ba8a-1a7861a5b665/kube-apiserver-operator/0.log 2026-03-09 19:19:31.994086 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_dcdc187f-6e3b-442c-80a1-e404ee5ebb9e/ 2026-03-09 19:19:31.994108 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_dcdc187f-6e3b-442c-80a1-e404ee5ebb9e/installer/ 2026-03-09 19:19:31.994129 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_dcdc187f-6e3b-442c-80a1-e404ee5ebb9e/installer/0.log 2026-03-09 19:19:31.994150 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/ 2026-03-09 19:19:31.994172 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-regeneration-controller/ 2026-03-09 19:19:31.994193 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-regeneration-controller/0.log 2026-03-09 19:19:31.994214 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-syncer/ 2026-03-09 19:19:31.994241 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-syncer/0.log 2026-03-09 19:19:31.994262 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-check-endpoints/ 2026-03-09 19:19:31.994284 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-check-endpoints/0.log 2026-03-09 19:19:31.994305 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-insecure-readyz/ 2026-03-09 19:19:31.994326 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-insecure-readyz/0.log 2026-03-09 19:19:31.994347 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver/ 2026-03-09 19:19:31.994368 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver/0.log 2026-03-09 19:19:31.994389 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/setup/ 2026-03-09 19:19:31.994410 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/setup/0.log 2026-03-09 19:19:31.994431 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_7e28616e-2c71-45e4-b93c-e76014c89d0d/ 2026-03-09 19:19:31.994452 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_7e28616e-2c71-45e4-b93c-e76014c89d0d/pruner/ 2026-03-09 19:19:31.994472 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_7e28616e-2c71-45e4-b93c-e76014c89d0d/pruner/0.log 2026-03-09 19:19:31.994492 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_f6987aec-8ee0-4026-99bb-a30b76e2b131/ 2026-03-09 19:19:31.994513 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_f6987aec-8ee0-4026-99bb-a30b76e2b131/pruner/ 2026-03-09 19:19:31.994535 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_f6987aec-8ee0-4026-99bb-a30b76e2b131/pruner/0.log 2026-03-09 19:19:31.994556 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-ctxcr_e4ce9992-733a-4ac6-ab14-610ac4ced250/ 2026-03-09 19:19:31.994577 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-ctxcr_e4ce9992-733a-4ac6-ab14-610ac4ced250/kube-controller-manager-operator/ 2026-03-09 19:19:31.994601 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-ctxcr_e4ce9992-733a-4ac6-ab14-610ac4ced250/kube-controller-manager-operator/0.log 2026-03-09 19:19:31.994630 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/ 2026-03-09 19:19:31.994662 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/ 2026-03-09 19:19:31.994685 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/0.log 2026-03-09 19:19:31.994707 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/1.log 2026-03-09 19:19:31.994733 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-cert-syncer/ 2026-03-09 19:19:31.994760 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-cert-syncer/0.log 2026-03-09 19:19:31.994783 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-recovery-controller/ 2026-03-09 19:19:31.994805 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-recovery-controller/0.log 2026-03-09 19:19:31.994827 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/ 2026-03-09 19:19:31.994849 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/1.log 2026-03-09 19:19:31.994871 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/2.log.gz 2026-03-09 19:19:31.994893 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_e3d6bf4f-253c-4694-ba14-62abd7d74285/ 2026-03-09 19:19:31.994915 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_e3d6bf4f-253c-4694-ba14-62abd7d74285/pruner/ 2026-03-09 19:19:31.994936 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_e3d6bf4f-253c-4694-ba14-62abd7d74285/pruner/0.log 2026-03-09 19:19:31.994958 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-46tk5_7227208b-b4f1-473c-9149-2a1c4d1cab32/ 2026-03-09 19:19:31.994980 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-46tk5_7227208b-b4f1-473c-9149-2a1c4d1cab32/kube-scheduler-operator-container/ 2026-03-09 19:19:31.995002 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-46tk5_7227208b-b4f1-473c-9149-2a1c4d1cab32/kube-scheduler-operator-container/0.log 2026-03-09 19:19:31.995050 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/ 2026-03-09 19:19:31.995075 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-cert-syncer/ 2026-03-09 19:19:31.995102 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-cert-syncer/0.log 2026-03-09 19:19:31.995125 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-recovery-controller/ 2026-03-09 19:19:31.995148 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-recovery-controller/0.log 2026-03-09 19:19:31.995169 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler/ 2026-03-09 19:19:31.995191 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler/0.log 2026-03-09 19:19:31.995212 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/wait-for-host-port/ 2026-03-09 19:19:31.995234 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/wait-for-host-port/0.log 2026-03-09 19:19:31.995350 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-gk67k_d87105c8-2398-44ec-b127-a2e30e767c1d/ 2026-03-09 19:19:31.995381 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-gk67k_d87105c8-2398-44ec-b127-a2e30e767c1d/kube-storage-version-migrator-operator/ 2026-03-09 19:19:31.995405 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-gk67k_d87105c8-2398-44ec-b127-a2e30e767c1d/kube-storage-version-migrator-operator/0.log 2026-03-09 19:19:31.995429 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-n5glb_2528c75b-c6dc-4347-b2e5-8279c1861c53/ 2026-03-09 19:19:31.995452 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-n5glb_2528c75b-c6dc-4347-b2e5-8279c1861c53/graceful-termination/ 2026-03-09 19:19:31.995474 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-n5glb_2528c75b-c6dc-4347-b2e5-8279c1861c53/graceful-termination/0.log 2026-03-09 19:19:31.995495 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-n5glb_2528c75b-c6dc-4347-b2e5-8279c1861c53/migrator/ 2026-03-09 19:19:31.995516 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-n5glb_2528c75b-c6dc-4347-b2e5-8279c1861c53/migrator/0.log 2026-03-09 19:19:31.995538 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-dbk5m_04e006fb-bb29-4683-b3a9-a17698564fa6/ 2026-03-09 19:19:31.995577 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-dbk5m_04e006fb-bb29-4683-b3a9-a17698564fa6/control-plane-machine-set-operator/ 2026-03-09 19:19:31.995599 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-dbk5m_04e006fb-bb29-4683-b3a9-a17698564fa6/control-plane-machine-set-operator/0.log 2026-03-09 19:19:31.995630 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-h8j2t_a6f5ce9b-e0f2-4bbb-9a18-7c62bdb830e4/ 2026-03-09 19:19:31.995652 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-h8j2t_a6f5ce9b-e0f2-4bbb-9a18-7c62bdb830e4/kube-rbac-proxy/ 2026-03-09 19:19:31.995674 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-h8j2t_a6f5ce9b-e0f2-4bbb-9a18-7c62bdb830e4/kube-rbac-proxy/0.log 2026-03-09 19:19:31.995695 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-h8j2t_a6f5ce9b-e0f2-4bbb-9a18-7c62bdb830e4/machine-api-operator/ 2026-03-09 19:19:31.995717 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-h8j2t_a6f5ce9b-e0f2-4bbb-9a18-7c62bdb830e4/machine-api-operator/0.log 2026-03-09 19:19:31.995739 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/ 2026-03-09 19:19:31.995760 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/kube-rbac-proxy-crio/ 2026-03-09 19:19:31.995781 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/kube-rbac-proxy-crio/0.log 2026-03-09 19:19:31.995802 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/setup/ 2026-03-09 19:19:31.995823 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/setup/0.log 2026-03-09 19:19:31.995845 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-ppt7p_1ec71021-0474-49ce-b545-4a973703b42b/ 2026-03-09 19:19:31.995865 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-ppt7p_1ec71021-0474-49ce-b545-4a973703b42b/kube-rbac-proxy/ 2026-03-09 19:19:31.995887 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-ppt7p_1ec71021-0474-49ce-b545-4a973703b42b/kube-rbac-proxy/0.log 2026-03-09 19:19:31.995908 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-ppt7p_1ec71021-0474-49ce-b545-4a973703b42b/machine-config-controller/ 2026-03-09 19:19:31.995930 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-ppt7p_1ec71021-0474-49ce-b545-4a973703b42b/machine-config-controller/0.log 2026-03-09 19:19:31.995951 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-kk7gs_3270571a-a484-4e66-8035-f43509b58add/ 2026-03-09 19:19:31.995971 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-kk7gs_3270571a-a484-4e66-8035-f43509b58add/kube-rbac-proxy/ 2026-03-09 19:19:31.995992 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-kk7gs_3270571a-a484-4e66-8035-f43509b58add/kube-rbac-proxy/0.log 2026-03-09 19:19:31.996048 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-kk7gs_3270571a-a484-4e66-8035-f43509b58add/machine-config-daemon/ 2026-03-09 19:19:31.996080 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-kk7gs_3270571a-a484-4e66-8035-f43509b58add/machine-config-daemon/10.log 2026-03-09 19:19:31.996105 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-kk7gs_3270571a-a484-4e66-8035-f43509b58add/machine-config-daemon/11.log 2026-03-09 19:19:31.996126 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-5ncxl_5275d8b9-8874-4c24-96b9-fdef4ef32d9b/ 2026-03-09 19:19:31.996147 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-5ncxl_5275d8b9-8874-4c24-96b9-fdef4ef32d9b/kube-rbac-proxy/ 2026-03-09 19:19:31.996168 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-5ncxl_5275d8b9-8874-4c24-96b9-fdef4ef32d9b/kube-rbac-proxy/0.log 2026-03-09 19:19:31.996190 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-5ncxl_5275d8b9-8874-4c24-96b9-fdef4ef32d9b/machine-config-operator/ 2026-03-09 19:19:31.996211 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-5ncxl_5275d8b9-8874-4c24-96b9-fdef4ef32d9b/machine-config-operator/0.log 2026-03-09 19:19:31.996233 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-sst54_d9788fbc-230b-4324-ba04-c706c0278411/ 2026-03-09 19:19:31.996254 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-sst54_d9788fbc-230b-4324-ba04-c706c0278411/machine-config-server/ 2026-03-09 19:19:31.996275 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-sst54_d9788fbc-230b-4324-ba04-c706c0278411/machine-config-server/0.log 2026-03-09 19:19:31.996296 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq_6dbcce1b-4861-49b4-aed4-aaa992fe1a79/ 2026-03-09 19:19:31.996317 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq_6dbcce1b-4861-49b4-aed4-aaa992fe1a79/extract/ 2026-03-09 19:19:31.996338 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq_6dbcce1b-4861-49b4-aed4-aaa992fe1a79/extract/0.log 2026-03-09 19:19:31.996359 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq_6dbcce1b-4861-49b4-aed4-aaa992fe1a79/pull/ 2026-03-09 19:19:31.996381 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq_6dbcce1b-4861-49b4-aed4-aaa992fe1a79/pull/0.log 2026-03-09 19:19:31.996403 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq_6dbcce1b-4861-49b4-aed4-aaa992fe1a79/util/ 2026-03-09 19:19:31.996425 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq_6dbcce1b-4861-49b4-aed4-aaa992fe1a79/util/0.log 2026-03-09 19:19:31.996452 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4_332c9a2e-4daa-4bc4-8020-1938abeccb55/ 2026-03-09 19:19:31.996475 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4_332c9a2e-4daa-4bc4-8020-1938abeccb55/extract/ 2026-03-09 19:19:31.996497 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4_332c9a2e-4daa-4bc4-8020-1938abeccb55/extract/0.log 2026-03-09 19:19:31.996520 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4_332c9a2e-4daa-4bc4-8020-1938abeccb55/pull/ 2026-03-09 19:19:31.996542 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4_332c9a2e-4daa-4bc4-8020-1938abeccb55/pull/0.log 2026-03-09 19:19:31.996563 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4_332c9a2e-4daa-4bc4-8020-1938abeccb55/util/ 2026-03-09 19:19:31.996585 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4_332c9a2e-4daa-4bc4-8020-1938abeccb55/util/0.log 2026-03-09 19:19:31.996607 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6_9a7665a2-307a-4f7f-939a-b93afc455415/ 2026-03-09 19:19:31.996629 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6_9a7665a2-307a-4f7f-939a-b93afc455415/extract/ 2026-03-09 19:19:31.996651 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6_9a7665a2-307a-4f7f-939a-b93afc455415/extract/0.log 2026-03-09 19:19:31.996672 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6_9a7665a2-307a-4f7f-939a-b93afc455415/pull/ 2026-03-09 19:19:31.996694 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6_9a7665a2-307a-4f7f-939a-b93afc455415/pull/0.log 2026-03-09 19:19:31.996717 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6_9a7665a2-307a-4f7f-939a-b93afc455415/util/ 2026-03-09 19:19:31.996738 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6_9a7665a2-307a-4f7f-939a-b93afc455415/util/0.log 2026-03-09 19:19:31.996761 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vhn27_df2155e5-7524-47f7-8c00-80c2ab292588/ 2026-03-09 19:19:31.996783 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vhn27_df2155e5-7524-47f7-8c00-80c2ab292588/extract-content/ 2026-03-09 19:19:31.996804 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vhn27_df2155e5-7524-47f7-8c00-80c2ab292588/extract-content/0.log 2026-03-09 19:19:31.996826 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vhn27_df2155e5-7524-47f7-8c00-80c2ab292588/extract-utilities/ 2026-03-09 19:19:31.996853 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vhn27_df2155e5-7524-47f7-8c00-80c2ab292588/extract-utilities/0.log 2026-03-09 19:19:31.996875 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vhn27_df2155e5-7524-47f7-8c00-80c2ab292588/registry-server/ 2026-03-09 19:19:31.996896 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vhn27_df2155e5-7524-47f7-8c00-80c2ab292588/registry-server/0.log.gz 2026-03-09 19:19:31.996918 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-29tmk_011ab61a-9a65-4112-8ab5-149d78479cc4/ 2026-03-09 19:19:31.996939 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-29tmk_011ab61a-9a65-4112-8ab5-149d78479cc4/extract-content/ 2026-03-09 19:19:31.996960 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-29tmk_011ab61a-9a65-4112-8ab5-149d78479cc4/extract-content/0.log 2026-03-09 19:19:31.996981 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-29tmk_011ab61a-9a65-4112-8ab5-149d78479cc4/extract-utilities/ 2026-03-09 19:19:31.997002 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-29tmk_011ab61a-9a65-4112-8ab5-149d78479cc4/extract-utilities/0.log 2026-03-09 19:19:31.997047 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-29tmk_011ab61a-9a65-4112-8ab5-149d78479cc4/registry-server/ 2026-03-09 19:19:31.997072 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-29tmk_011ab61a-9a65-4112-8ab5-149d78479cc4/registry-server/0.log.gz 2026-03-09 19:19:31.997098 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j_7112cff8-f71e-4537-853f-155cfd48f5b6/ 2026-03-09 19:19:31.997123 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j_7112cff8-f71e-4537-853f-155cfd48f5b6/extract/ 2026-03-09 19:19:31.997145 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j_7112cff8-f71e-4537-853f-155cfd48f5b6/extract/0.log 2026-03-09 19:19:31.997167 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j_7112cff8-f71e-4537-853f-155cfd48f5b6/pull/ 2026-03-09 19:19:31.997188 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j_7112cff8-f71e-4537-853f-155cfd48f5b6/pull/0.log 2026-03-09 19:19:31.997209 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j_7112cff8-f71e-4537-853f-155cfd48f5b6/util/ 2026-03-09 19:19:31.997230 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j_7112cff8-f71e-4537-853f-155cfd48f5b6/util/0.log 2026-03-09 19:19:31.997251 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-556c4_872fb4be-c421-4274-8646-56e708f8c698/ 2026-03-09 19:19:31.997281 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-556c4_872fb4be-c421-4274-8646-56e708f8c698/marketplace-operator/ 2026-03-09 19:19:31.997359 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-556c4_872fb4be-c421-4274-8646-56e708f8c698/marketplace-operator/0.log 2026-03-09 19:19:31.997383 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-pchlh_6a1328a9-ebc5-4976-8ed0-45de86204b20/ 2026-03-09 19:19:31.997406 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-pchlh_6a1328a9-ebc5-4976-8ed0-45de86204b20/extract-content/ 2026-03-09 19:19:31.997428 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-pchlh_6a1328a9-ebc5-4976-8ed0-45de86204b20/extract-content/0.log 2026-03-09 19:19:31.997449 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-pchlh_6a1328a9-ebc5-4976-8ed0-45de86204b20/extract-utilities/ 2026-03-09 19:19:31.997471 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-pchlh_6a1328a9-ebc5-4976-8ed0-45de86204b20/extract-utilities/0.log 2026-03-09 19:19:31.997493 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-pchlh_6a1328a9-ebc5-4976-8ed0-45de86204b20/registry-server/ 2026-03-09 19:19:31.997514 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-pchlh_6a1328a9-ebc5-4976-8ed0-45de86204b20/registry-server/0.log 2026-03-09 19:19:31.997536 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-tc2f5_f077b409-1e21-4fb0-a973-8c57822d2b94/ 2026-03-09 19:19:31.997558 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-tc2f5_f077b409-1e21-4fb0-a973-8c57822d2b94/extract-content/ 2026-03-09 19:19:31.997579 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-tc2f5_f077b409-1e21-4fb0-a973-8c57822d2b94/extract-content/0.log 2026-03-09 19:19:31.997600 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-tc2f5_f077b409-1e21-4fb0-a973-8c57822d2b94/extract-utilities/ 2026-03-09 19:19:31.997621 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-tc2f5_f077b409-1e21-4fb0-a973-8c57822d2b94/extract-utilities/0.log 2026-03-09 19:19:31.997643 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-tc2f5_f077b409-1e21-4fb0-a973-8c57822d2b94/registry-server/ 2026-03-09 19:19:31.997664 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-tc2f5_f077b409-1e21-4fb0-a973-8c57822d2b94/registry-server/0.log.gz 2026-03-09 19:19:31.997685 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/ 2026-03-09 19:19:31.997707 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/bond-cni-plugin/ 2026-03-09 19:19:31.997728 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/bond-cni-plugin/0.log 2026-03-09 19:19:31.997749 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/cni-plugins/ 2026-03-09 19:19:31.997775 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/cni-plugins/0.log 2026-03-09 19:19:31.997797 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/egress-router-binary-copy/ 2026-03-09 19:19:31.997818 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/egress-router-binary-copy/0.log 2026-03-09 19:19:31.997839 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/kube-multus-additional-cni-plugins/ 2026-03-09 19:19:31.997860 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/kube-multus-additional-cni-plugins/0.log 2026-03-09 19:19:31.997881 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/routeoverride-cni/ 2026-03-09 19:19:31.997902 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/routeoverride-cni/0.log 2026-03-09 19:19:31.997923 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/whereabouts-cni-bincopy/ 2026-03-09 19:19:31.997944 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/whereabouts-cni-bincopy/0.log 2026-03-09 19:19:31.997965 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/whereabouts-cni/ 2026-03-09 19:19:31.997986 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-b9gd4_84199f52-999d-4a44-91c7-a343ba59b10d/whereabouts-cni/0.log 2026-03-09 19:19:31.998007 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-d9kvs_9016680a-98b9-4503-a9d6-251355aaecc3/ 2026-03-09 19:19:31.998054 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-d9kvs_9016680a-98b9-4503-a9d6-251355aaecc3/kube-rbac-proxy/ 2026-03-09 19:19:31.998079 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-d9kvs_9016680a-98b9-4503-a9d6-251355aaecc3/kube-rbac-proxy/0.log 2026-03-09 19:19:31.998101 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-d9kvs_9016680a-98b9-4503-a9d6-251355aaecc3/multus-admission-controller/ 2026-03-09 19:19:31.998123 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-d9kvs_9016680a-98b9-4503-a9d6-251355aaecc3/multus-admission-controller/0.log 2026-03-09 19:19:31.998145 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-lw2hk_1a255bc9-2034-4a34-8240-f1fd42e808bd/ 2026-03-09 19:19:31.998168 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-lw2hk_1a255bc9-2034-4a34-8240-f1fd42e808bd/kube-multus/ 2026-03-09 19:19:31.998190 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-lw2hk_1a255bc9-2034-4a34-8240-f1fd42e808bd/kube-multus/0.log 2026-03-09 19:19:31.998217 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-lw2hk_1a255bc9-2034-4a34-8240-f1fd42e808bd/kube-multus/1.log 2026-03-09 19:19:31.998240 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lf7bd_9ac2c88b-a0bc-482c-90fa-165d30f045e8/ 2026-03-09 19:19:31.998261 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lf7bd_9ac2c88b-a0bc-482c-90fa-165d30f045e8/kube-rbac-proxy/ 2026-03-09 19:19:31.998283 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lf7bd_9ac2c88b-a0bc-482c-90fa-165d30f045e8/kube-rbac-proxy/0.log 2026-03-09 19:19:31.998305 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lf7bd_9ac2c88b-a0bc-482c-90fa-165d30f045e8/network-metrics-daemon/ 2026-03-09 19:19:31.998326 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lf7bd_9ac2c88b-a0bc-482c-90fa-165d30f045e8/network-metrics-daemon/0.log 2026-03-09 19:19:31.998348 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-b86cm_must-gather-8fp4b_480bea75-1d63-4af0-b2e2-b7bf9d804872/ 2026-03-09 19:19:31.998370 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-b86cm_must-gather-8fp4b_480bea75-1d63-4af0-b2e2-b7bf9d804872/copy/ 2026-03-09 19:19:31.998393 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-b86cm_must-gather-8fp4b_480bea75-1d63-4af0-b2e2-b7bf9d804872/gather/ 2026-03-09 19:19:31.998415 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/ 2026-03-09 19:19:31.998437 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/networking-console-plugin/ 2026-03-09 19:19:31.998459 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/networking-console-plugin/0.log 2026-03-09 19:19:31.998481 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/ 2026-03-09 19:19:31.998503 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/ 2026-03-09 19:19:31.998525 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/0.log 2026-03-09 19:19:31.998547 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/ 2026-03-09 19:19:31.998569 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/network-check-target-container/ 2026-03-09 19:19:31.998591 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/network-check-target-container/0.log 2026-03-09 19:19:31.998613 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/ 2026-03-09 19:19:31.998639 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/approver/ 2026-03-09 19:19:31.998662 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/approver/0.log 2026-03-09 19:19:31.998684 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/webhook/ 2026-03-09 19:19:31.998706 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/webhook/0.log 2026-03-09 19:19:31.998728 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/ 2026-03-09 19:19:31.998748 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/iptables-alerter/ 2026-03-09 19:19:31.998770 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/iptables-alerter/0.log 2026-03-09 19:19:31.998791 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/ 2026-03-09 19:19:31.998812 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/network-operator/ 2026-03-09 19:19:31.998833 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/network-operator/0.log 2026-03-09 19:19:31.998855 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-5dcbbd79cf-sk2qd_be8695e5-622f-41f2-af2e-bd194fdefeb9/ 2026-03-09 19:19:31.998875 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-5dcbbd79cf-sk2qd_be8695e5-622f-41f2-af2e-bd194fdefeb9/nmstate-console-plugin/ 2026-03-09 19:19:31.998897 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-5dcbbd79cf-sk2qd_be8695e5-622f-41f2-af2e-bd194fdefeb9/nmstate-console-plugin/0.log 2026-03-09 19:19:31.998917 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-msftq_3a2bd74c-644c-4c41-9159-5c8eadc45763/ 2026-03-09 19:19:31.998939 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-msftq_3a2bd74c-644c-4c41-9159-5c8eadc45763/nmstate-handler/ 2026-03-09 19:19:31.998959 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-msftq_3a2bd74c-644c-4c41-9159-5c8eadc45763/nmstate-handler/0.log 2026-03-09 19:19:31.998980 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-69594cc75-g2mff_f003e733-9aab-493c-ad84-3b6ec8bae6ee/ 2026-03-09 19:19:31.999000 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-69594cc75-g2mff_f003e733-9aab-493c-ad84-3b6ec8bae6ee/kube-rbac-proxy/ 2026-03-09 19:19:31.999044 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-69594cc75-g2mff_f003e733-9aab-493c-ad84-3b6ec8bae6ee/kube-rbac-proxy/0.log 2026-03-09 19:19:31.999069 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-69594cc75-g2mff_f003e733-9aab-493c-ad84-3b6ec8bae6ee/nmstate-metrics/ 2026-03-09 19:19:31.999099 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-69594cc75-g2mff_f003e733-9aab-493c-ad84-3b6ec8bae6ee/nmstate-metrics/0.log 2026-03-09 19:19:31.999124 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-75c5dccd6c-2j8gv_e720485c-7121-43fb-aa59-e383aad4c545/ 2026-03-09 19:19:31.999145 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-75c5dccd6c-2j8gv_e720485c-7121-43fb-aa59-e383aad4c545/nmstate-operator/ 2026-03-09 19:19:31.999166 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-75c5dccd6c-2j8gv_e720485c-7121-43fb-aa59-e383aad4c545/nmstate-operator/0.log 2026-03-09 19:19:31.999187 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-786f45cff4-hnr59_5857c061-39ca-4cdf-a64f-b2c5e60c6a35/ 2026-03-09 19:19:31.999209 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-786f45cff4-hnr59_5857c061-39ca-4cdf-a64f-b2c5e60c6a35/nmstate-webhook/ 2026-03-09 19:19:31.999230 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-786f45cff4-hnr59_5857c061-39ca-4cdf-a64f-b2c5e60c6a35/nmstate-webhook/0.log 2026-03-09 19:19:31.999251 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-bsmz7_aca75c59-dc3a-4bd4-aeed-4ee5c715e5f6/ 2026-03-09 19:19:31.999272 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-bsmz7_aca75c59-dc3a-4bd4-aeed-4ee5c715e5f6/fix-audit-permissions/ 2026-03-09 19:19:31.999294 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-bsmz7_aca75c59-dc3a-4bd4-aeed-4ee5c715e5f6/fix-audit-permissions/0.log 2026-03-09 19:19:31.999315 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-bsmz7_aca75c59-dc3a-4bd4-aeed-4ee5c715e5f6/oauth-apiserver/ 2026-03-09 19:19:31.999338 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-bsmz7_aca75c59-dc3a-4bd4-aeed-4ee5c715e5f6/oauth-apiserver/0.log 2026-03-09 19:19:31.999366 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-jrc42_d2d34b0b-073c-47cb-9c2c-e2863dc06c23/ 2026-03-09 19:19:31.999391 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-jrc42_d2d34b0b-073c-47cb-9c2c-e2863dc06c23/catalog-operator/ 2026-03-09 19:19:31.999415 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-jrc42_d2d34b0b-073c-47cb-9c2c-e2863dc06c23/catalog-operator/0.log.gz 2026-03-09 19:19:31.999440 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551350-l7kv7_9b2d4a49-67a2-4a60-98ac-a10446691d92/ 2026-03-09 19:19:31.999584 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551350-l7kv7_9b2d4a49-67a2-4a60-98ac-a10446691d92/collect-profiles/ 2026-03-09 19:19:31.999622 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551350-l7kv7_9b2d4a49-67a2-4a60-98ac-a10446691d92/collect-profiles/0.log 2026-03-09 19:19:31.999650 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551365-4nf9z_16e14459-01b6-4c39-96e8-9e24d5293791/ 2026-03-09 19:19:31.999680 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551365-4nf9z_16e14459-01b6-4c39-96e8-9e24d5293791/collect-profiles/ 2026-03-09 19:19:31.999703 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551365-4nf9z_16e14459-01b6-4c39-96e8-9e24d5293791/collect-profiles/0.log 2026-03-09 19:19:31.999727 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551380-c2m5h_3625a79c-d381-4f4d-ae55-348a14977ca8/ 2026-03-09 19:19:31.999750 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551380-c2m5h_3625a79c-d381-4f4d-ae55-348a14977ca8/collect-profiles/ 2026-03-09 19:19:31.999773 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551380-c2m5h_3625a79c-d381-4f4d-ae55-348a14977ca8/collect-profiles/0.log 2026-03-09 19:19:31.999795 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551395-rr5fb_fc328dcf-ed65-4f2c-94ba-f4b8f5da59a3/ 2026-03-09 19:19:31.999817 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551395-rr5fb_fc328dcf-ed65-4f2c-94ba-f4b8f5da59a3/collect-profiles/ 2026-03-09 19:19:31.999840 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29551395-rr5fb_fc328dcf-ed65-4f2c-94ba-f4b8f5da59a3/collect-profiles/0.log 2026-03-09 19:19:31.999862 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-x76nr_43d18118-9a44-4b09-add9-7df52470e1c7/ 2026-03-09 19:19:31.999883 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-x76nr_43d18118-9a44-4b09-add9-7df52470e1c7/olm-operator/ 2026-03-09 19:19:31.999905 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-x76nr_43d18118-9a44-4b09-add9-7df52470e1c7/olm-operator/0.log 2026-03-09 19:19:31.999927 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-ft9v6_7dfd5d64-f6dc-40bd-83d1-57e685cd4535/ 2026-03-09 19:19:31.999949 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-ft9v6_7dfd5d64-f6dc-40bd-83d1-57e685cd4535/kube-rbac-proxy/ 2026-03-09 19:19:31.999971 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-ft9v6_7dfd5d64-f6dc-40bd-83d1-57e685cd4535/kube-rbac-proxy/0.log 2026-03-09 19:19:31.999992 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-ft9v6_7dfd5d64-f6dc-40bd-83d1-57e685cd4535/package-server-manager/ 2026-03-09 19:19:32.000033 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-ft9v6_7dfd5d64-f6dc-40bd-83d1-57e685cd4535/package-server-manager/0.log 2026-03-09 19:19:32.000064 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-2lkq2_1722e733-725b-4985-8365-0f8f3ad0d10d/ 2026-03-09 19:19:32.000088 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-2lkq2_1722e733-725b-4985-8365-0f8f3ad0d10d/packageserver/ 2026-03-09 19:19:32.000116 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-2lkq2_1722e733-725b-4985-8365-0f8f3ad0d10d/packageserver/0.log 2026-03-09 19:19:32.000140 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-cqfq5_85b873a4-96da-407a-b4af-30ba3aa97519/ 2026-03-09 19:19:32.000163 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-cqfq5_85b873a4-96da-407a-b4af-30ba3aa97519/prometheus-operator/ 2026-03-09 19:19:32.000185 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-cqfq5_85b873a4-96da-407a-b4af-30ba3aa97519/prometheus-operator/0.log 2026-03-09 19:19:32.000207 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-559887c586-44lfv_c294d09f-af0a-400e-90ea-1097080fb096/ 2026-03-09 19:19:32.000228 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-559887c586-44lfv_c294d09f-af0a-400e-90ea-1097080fb096/prometheus-operator-admission-webhook/ 2026-03-09 19:19:32.000250 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-559887c586-44lfv_c294d09f-af0a-400e-90ea-1097080fb096/prometheus-operator-admission-webhook/0.log 2026-03-09 19:19:32.000272 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-559887c586-xfmsd_80d114e5-b1d1-496c-a0c1-3eeb8d2f67c8/ 2026-03-09 19:19:32.000295 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-559887c586-xfmsd_80d114e5-b1d1-496c-a0c1-3eeb8d2f67c8/prometheus-operator-admission-webhook/ 2026-03-09 19:19:32.000318 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-559887c586-xfmsd_80d114e5-b1d1-496c-a0c1-3eeb8d2f67c8/prometheus-operator-admission-webhook/0.log 2026-03-09 19:19:32.000340 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-j8xx6_52b04c6b-da35-4f2a-a5f2-06370a59da78/ 2026-03-09 19:19:32.000362 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-j8xx6_52b04c6b-da35-4f2a-a5f2-06370a59da78/operator/ 2026-03-09 19:19:32.000385 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-j8xx6_52b04c6b-da35-4f2a-a5f2-06370a59da78/operator/0.log 2026-03-09 19:19:32.000407 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-hkkkk_ce0d9e34-5f6c-4503-95a0-6a127c905bee/ 2026-03-09 19:19:32.000429 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-hkkkk_ce0d9e34-5f6c-4503-95a0-6a127c905bee/observability-ui-dashboards/ 2026-03-09 19:19:32.000451 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-hkkkk_ce0d9e34-5f6c-4503-95a0-6a127c905bee/observability-ui-dashboards/0.log 2026-03-09 19:19:32.000473 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-p679h_94baa4ca-adf1-461f-a309-a1639aafd708/ 2026-03-09 19:19:32.000495 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-p679h_94baa4ca-adf1-461f-a309-a1639aafd708/perses-operator/ 2026-03-09 19:19:32.000521 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-p679h_94baa4ca-adf1-461f-a309-a1639aafd708/perses-operator/0.log 2026-03-09 19:19:32.000544 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-d6g54_e896f92d-7d30-4f36-b892-5c8c9c792530/ 2026-03-09 19:19:32.000566 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-d6g54_e896f92d-7d30-4f36-b892-5c8c9c792530/kube-rbac-proxy/ 2026-03-09 19:19:32.000587 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-d6g54_e896f92d-7d30-4f36-b892-5c8c9c792530/kube-rbac-proxy/0.log 2026-03-09 19:19:32.000609 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-d6g54_e896f92d-7d30-4f36-b892-5c8c9c792530/ovnkube-cluster-manager/ 2026-03-09 19:19:32.000630 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-d6g54_e896f92d-7d30-4f36-b892-5c8c9c792530/ovnkube-cluster-manager/0.log 2026-03-09 19:19:32.000651 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/ 2026-03-09 19:19:32.000672 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/kube-rbac-proxy-node/ 2026-03-09 19:19:32.000694 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/kube-rbac-proxy-node/0.log 2026-03-09 19:19:32.000716 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/kube-rbac-proxy-ovn-metrics/ 2026-03-09 19:19:32.000737 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/kube-rbac-proxy-ovn-metrics/0.log 2026-03-09 19:19:32.000758 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/kubecfg-setup/ 2026-03-09 19:19:32.000779 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/kubecfg-setup/0.log 2026-03-09 19:19:32.000801 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/nbdb/ 2026-03-09 19:19:32.000822 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/nbdb/0.log 2026-03-09 19:19:32.000843 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/northd/ 2026-03-09 19:19:32.000864 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/northd/0.log 2026-03-09 19:19:32.000885 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/ovn-acl-logging/ 2026-03-09 19:19:32.000906 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/ovn-acl-logging/0.log 2026-03-09 19:19:32.000932 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/ovn-controller/ 2026-03-09 19:19:32.000953 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/ovn-controller/0.log 2026-03-09 19:19:32.000974 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/ovnkube-controller/ 2026-03-09 19:19:32.000994 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/ovnkube-controller/0.log.gz 2026-03-09 19:19:32.001035 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/sbdb/ 2026-03-09 19:19:32.001065 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-hl2fd_20e65f74-ecab-4bad-b2ea-09c0fac9406d/sbdb/0.log 2026-03-09 19:19:32.001087 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-777997c889-676zh_cc8e8bc1-27d6-41ee-a3ad-5f54da201888/ 2026-03-09 19:19:32.001110 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-777997c889-676zh_cc8e8bc1-27d6-41ee-a3ad-5f54da201888/route-controller-manager/ 2026-03-09 19:19:32.001131 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-777997c889-676zh_cc8e8bc1-27d6-41ee-a3ad-5f54da201888/route-controller-manager/0.log 2026-03-09 19:19:32.001153 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-hh9sf_243465ec-ca31-4ec1-b5ca-1e1318f37c16/ 2026-03-09 19:19:32.001175 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-hh9sf_243465ec-ca31-4ec1-b5ca-1e1318f37c16/service-ca-operator/ 2026-03-09 19:19:32.001197 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-hh9sf_243465ec-ca31-4ec1-b5ca-1e1318f37c16/service-ca-operator/0.log 2026-03-09 19:19:32.001294 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-4pwqq_ba8c991f-dcb9-4206-ad42-dedc0f6d04cb/ 2026-03-09 19:19:32.001335 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-4pwqq_ba8c991f-dcb9-4206-ad42-dedc0f6d04cb/service-ca-controller/ 2026-03-09 19:19:32.001361 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-4pwqq_ba8c991f-dcb9-4206-ad42-dedc0f6d04cb/service-ca-controller/0.log 2026-03-09 19:19:32.001385 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm_7022fc4e-6faf-4abb-9677-963728a8d91d/ 2026-03-09 19:19:32.001409 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm_7022fc4e-6faf-4abb-9677-963728a8d91d/extract/ 2026-03-09 19:19:32.001433 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm_7022fc4e-6faf-4abb-9677-963728a8d91d/extract/0.log 2026-03-09 19:19:32.001456 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm_7022fc4e-6faf-4abb-9677-963728a8d91d/pull/ 2026-03-09 19:19:32.001486 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm_7022fc4e-6faf-4abb-9677-963728a8d91d/pull/0.log 2026-03-09 19:19:32.001510 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm_7022fc4e-6faf-4abb-9677-963728a8d91d/util/ 2026-03-09 19:19:32.001536 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm_7022fc4e-6faf-4abb-9677-963728a8d91d/util/0.log 2026-03-09 19:19:32.001560 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx_175451d8-941f-4b65-a51c-60ec0d7427d1/ 2026-03-09 19:19:32.001583 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx_175451d8-941f-4b65-a51c-60ec0d7427d1/extract/ 2026-03-09 19:19:32.001606 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx_175451d8-941f-4b65-a51c-60ec0d7427d1/extract/0.log 2026-03-09 19:19:32.001629 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx_175451d8-941f-4b65-a51c-60ec0d7427d1/pull/ 2026-03-09 19:19:32.001651 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx_175451d8-941f-4b65-a51c-60ec0d7427d1/pull/0.log 2026-03-09 19:19:32.001674 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx_175451d8-941f-4b65-a51c-60ec0d7427d1/util/ 2026-03-09 19:19:32.001696 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx_175451d8-941f-4b65-a51c-60ec0d7427d1/util/0.log 2026-03-09 19:19:32.001718 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-6db6876945-9vg4l_9eb96ad1-a011-482f-bbdd-edfd673217b5/ 2026-03-09 19:19:32.001740 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-6db6876945-9vg4l_9eb96ad1-a011-482f-bbdd-edfd673217b5/manager/ 2026-03-09 19:19:32.001762 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-6db6876945-9vg4l_9eb96ad1-a011-482f-bbdd-edfd673217b5/manager/0.log.gz 2026-03-09 19:19:32.001784 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-55d77d7b5c-cjvgb_0a1af309-4a43-4d58-8912-abc1ed1e626a/ 2026-03-09 19:19:32.001807 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-55d77d7b5c-cjvgb_0a1af309-4a43-4d58-8912-abc1ed1e626a/manager/ 2026-03-09 19:19:32.001829 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-55d77d7b5c-cjvgb_0a1af309-4a43-4d58-8912-abc1ed1e626a/manager/0.log.gz 2026-03-09 19:19:32.001851 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-5d87c9d997-5rbnb_bb4823b7-c205-41c0-ba4d-d909ad9ff9cb/ 2026-03-09 19:19:32.001873 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-5d87c9d997-5rbnb_bb4823b7-c205-41c0-ba4d-d909ad9ff9cb/manager/ 2026-03-09 19:19:32.001899 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-5d87c9d997-5rbnb_bb4823b7-c205-41c0-ba4d-d909ad9ff9cb/manager/0.log 2026-03-09 19:19:32.001921 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-64db6967f8-hjztj_7507717c-322f-43de-88ba-fc79b6a5a3f0/ 2026-03-09 19:19:32.001943 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-64db6967f8-hjztj_7507717c-322f-43de-88ba-fc79b6a5a3f0/manager/ 2026-03-09 19:19:32.001964 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-64db6967f8-hjztj_7507717c-322f-43de-88ba-fc79b6a5a3f0/manager/0.log 2026-03-09 19:19:32.001986 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-cf99c678f-k4q8b_0b492a45-c917-4c00-abef-13abf40e71d1/ 2026-03-09 19:19:32.002008 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-cf99c678f-k4q8b_0b492a45-c917-4c00-abef-13abf40e71d1/manager/ 2026-03-09 19:19:32.002056 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-cf99c678f-k4q8b_0b492a45-c917-4c00-abef-13abf40e71d1/manager/0.log 2026-03-09 19:19:32.002080 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-78bc7f9bd9-vhljc_100889e4-2f00-4685-a5a7-6f9b73bb343f/ 2026-03-09 19:19:32.002102 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-78bc7f9bd9-vhljc_100889e4-2f00-4685-a5a7-6f9b73bb343f/manager/ 2026-03-09 19:19:32.002124 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-78bc7f9bd9-vhljc_100889e4-2f00-4685-a5a7-6f9b73bb343f/manager/0.log 2026-03-09 19:19:32.002145 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-f7fcc58b9-rldv2_c9d3c230-c74c-4cc4-af9f-f23fd5d9557c/ 2026-03-09 19:19:32.002167 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-f7fcc58b9-rldv2_c9d3c230-c74c-4cc4-af9f-f23fd5d9557c/manager/ 2026-03-09 19:19:32.002189 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-f7fcc58b9-rldv2_c9d3c230-c74c-4cc4-af9f-f23fd5d9557c/manager/0.log 2026-03-09 19:19:32.002210 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-545456dc4-wzvf8_772511ff-89ac-4190-8142-3bf3e4ef8423/ 2026-03-09 19:19:32.002232 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-545456dc4-wzvf8_772511ff-89ac-4190-8142-3bf3e4ef8423/manager/ 2026-03-09 19:19:32.002253 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-545456dc4-wzvf8_772511ff-89ac-4190-8142-3bf3e4ef8423/manager/0.log 2026-03-09 19:19:32.002275 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-7c789f89c6-pc4fs_71c62d87-8310-4ebd-8449-df18a56dc391/ 2026-03-09 19:19:32.002296 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-7c789f89c6-pc4fs_71c62d87-8310-4ebd-8449-df18a56dc391/manager/ 2026-03-09 19:19:32.002321 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-7c789f89c6-pc4fs_71c62d87-8310-4ebd-8449-df18a56dc391/manager/0.log 2026-03-09 19:19:32.002343 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-67d996989d-dhq9j_d878ceb7-5af9-4a91-82cb-ed03b73f1b1d/ 2026-03-09 19:19:32.002364 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-67d996989d-dhq9j_d878ceb7-5af9-4a91-82cb-ed03b73f1b1d/manager/ 2026-03-09 19:19:32.002385 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-67d996989d-dhq9j_d878ceb7-5af9-4a91-82cb-ed03b73f1b1d/manager/0.log 2026-03-09 19:19:32.002406 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-7b6bfb6475-9jwph_b10f4933-a23d-4c0b-9834-40caa60b158c/ 2026-03-09 19:19:32.002427 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-7b6bfb6475-9jwph_b10f4933-a23d-4c0b-9834-40caa60b158c/manager/ 2026-03-09 19:19:32.002449 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-7b6bfb6475-9jwph_b10f4933-a23d-4c0b-9834-40caa60b158c/manager/0.log 2026-03-09 19:19:32.002470 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-54688575f-974k8_89a79a12-ce90-47f7-b0c4-c0976d7a4b1f/ 2026-03-09 19:19:32.002491 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-54688575f-974k8_89a79a12-ce90-47f7-b0c4-c0976d7a4b1f/manager/ 2026-03-09 19:19:32.002512 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-54688575f-974k8_89a79a12-ce90-47f7-b0c4-c0976d7a4b1f/manager/0.log 2026-03-09 19:19:32.002533 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-74b6b5dc96-qz976_d6f3f569-2d6b-4c06-a814-de946397de51/ 2026-03-09 19:19:32.002555 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-74b6b5dc96-qz976_d6f3f569-2d6b-4c06-a814-de946397de51/manager/ 2026-03-09 19:19:32.002576 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-74b6b5dc96-qz976_d6f3f569-2d6b-4c06-a814-de946397de51/manager/0.log 2026-03-09 19:19:32.002597 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5d86c7ddb7-6864w_d1eba3e1-a741-4ca6-a97e-c42565f64d2b/ 2026-03-09 19:19:32.002618 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5d86c7ddb7-6864w_d1eba3e1-a741-4ca6-a97e-c42565f64d2b/manager/ 2026-03-09 19:19:32.002640 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5d86c7ddb7-6864w_d1eba3e1-a741-4ca6-a97e-c42565f64d2b/manager/0.log 2026-03-09 19:19:32.002662 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-7c6767dc9cp28hq_212b84ba-bcda-4820-8388-7d2ef286b7a1/ 2026-03-09 19:19:32.002684 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-7c6767dc9cp28hq_212b84ba-bcda-4820-8388-7d2ef286b7a1/manager/ 2026-03-09 19:19:32.002706 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-7c6767dc9cp28hq_212b84ba-bcda-4820-8388-7d2ef286b7a1/manager/0.log 2026-03-09 19:19:32.002733 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-64797568c9-7qbhc_9162d85f-f6f9-4a12-8511-d11676a6398a/ 2026-03-09 19:19:32.002755 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-64797568c9-7qbhc_9162d85f-f6f9-4a12-8511-d11676a6398a/manager/ 2026-03-09 19:19:32.002777 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-64797568c9-7qbhc_9162d85f-f6f9-4a12-8511-d11676a6398a/manager/0.log.gz 2026-03-09 19:19:32.002798 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-cjjbr_43193e76-c853-4bc6-89e4-12ff09c8fbcb/ 2026-03-09 19:19:32.002820 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-cjjbr_43193e76-c853-4bc6-89e4-12ff09c8fbcb/registry-server/ 2026-03-09 19:19:32.002842 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-cjjbr_43193e76-c853-4bc6-89e4-12ff09c8fbcb/registry-server/0.log 2026-03-09 19:19:32.002863 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-75684d597f-w9h2t_6bc651e4-1359-43b1-bc53-1a561195cf4a/ 2026-03-09 19:19:32.002884 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-75684d597f-w9h2t_6bc651e4-1359-43b1-bc53-1a561195cf4a/manager/ 2026-03-09 19:19:32.002905 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-75684d597f-w9h2t_6bc651e4-1359-43b1-bc53-1a561195cf4a/manager/0.log 2026-03-09 19:19:32.002926 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-648564c9fc-wbjvw_d498150e-134b-4359-92c6-300b8fbe3b1a/ 2026-03-09 19:19:32.002947 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-648564c9fc-wbjvw_d498150e-134b-4359-92c6-300b8fbe3b1a/manager/ 2026-03-09 19:19:32.002969 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-648564c9fc-wbjvw_d498150e-134b-4359-92c6-300b8fbe3b1a/manager/0.log 2026-03-09 19:19:32.002991 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-xzdll_172ecee8-2a7b-4e13-b095-ca2a442932d2/ 2026-03-09 19:19:32.003032 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-xzdll_172ecee8-2a7b-4e13-b095-ca2a442932d2/operator/ 2026-03-09 19:19:32.003062 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-xzdll_172ecee8-2a7b-4e13-b095-ca2a442932d2/operator/0.log 2026-03-09 19:19:32.003085 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-9b9ff9f4d-k9s84_528fcc81-e85c-4764-9413-3957ba8c6fd2/ 2026-03-09 19:19:32.003107 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-9b9ff9f4d-k9s84_528fcc81-e85c-4764-9413-3957ba8c6fd2/manager/ 2026-03-09 19:19:32.003129 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-9b9ff9f4d-k9s84_528fcc81-e85c-4764-9413-3957ba8c6fd2/manager/0.log 2026-03-09 19:19:32.003156 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-5fdb694969-zffr5_28a07a44-f359-40b3-a2d4-850cb3822cb4/ 2026-03-09 19:19:32.003178 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-5fdb694969-zffr5_28a07a44-f359-40b3-a2d4-850cb3822cb4/manager/ 2026-03-09 19:19:32.003200 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-5fdb694969-zffr5_28a07a44-f359-40b3-a2d4-850cb3822cb4/manager/0.log.gz 2026-03-09 19:19:32.003221 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-55b5ff4dbb-z5d4k_2e967d7a-a1cf-44b9-ae66-62c4c5c81b55/ 2026-03-09 19:19:32.003243 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-55b5ff4dbb-z5d4k_2e967d7a-a1cf-44b9-ae66-62c4c5c81b55/manager/ 2026-03-09 19:19:32.003263 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-55b5ff4dbb-z5d4k_2e967d7a-a1cf-44b9-ae66-62c4c5c81b55/manager/0.log 2026-03-09 19:19:32.003284 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-85b655bd8f-llgvv_3e660422-3d8e-4716-b1df-6aa0d193e8f6/ 2026-03-09 19:19:32.003306 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-85b655bd8f-llgvv_3e660422-3d8e-4716-b1df-6aa0d193e8f6/manager/ 2026-03-09 19:19:32.003328 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-85b655bd8f-llgvv_3e660422-3d8e-4716-b1df-6aa0d193e8f6/manager/0.log.gz 2026-03-09 19:19:32.003350 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-snmkh_81e65025-6a00-4e95-83fb-ccf57455d09e/ 2026-03-09 19:19:32.003373 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-snmkh_81e65025-6a00-4e95-83fb-ccf57455d09e/registry-server/ 2026-03-09 19:19:32.003396 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-snmkh_81e65025-6a00-4e95-83fb-ccf57455d09e/registry-server/0.log 2026-03-09 19:19:32.003418 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_fa689f50-deca-4456-946b-edd730385d48/ 2026-03-09 19:19:32.003441 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_fa689f50-deca-4456-946b-edd730385d48/alertmanager/ 2026-03-09 19:19:32.003462 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_fa689f50-deca-4456-946b-edd730385d48/alertmanager/0.log 2026-03-09 19:19:32.003484 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_fa689f50-deca-4456-946b-edd730385d48/config-reloader/ 2026-03-09 19:19:32.003505 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_fa689f50-deca-4456-946b-edd730385d48/config-reloader/0.log 2026-03-09 19:19:32.003527 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_fa689f50-deca-4456-946b-edd730385d48/init-config-reloader/ 2026-03-09 19:19:32.003562 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_fa689f50-deca-4456-946b-edd730385d48/init-config-reloader/0.log 2026-03-09 19:19:32.003590 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/ 2026-03-09 19:19:32.003612 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/ceilometer-central-agent/ 2026-03-09 19:19:32.003634 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/ceilometer-central-agent/0.log 2026-03-09 19:19:32.003656 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/ceilometer-notification-agent/ 2026-03-09 19:19:32.003678 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/ceilometer-notification-agent/0.log 2026-03-09 19:19:32.003699 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/proxy-httpd/ 2026-03-09 19:19:32.003720 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/proxy-httpd/0.log 2026-03-09 19:19:32.003741 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/sg-core/ 2026-03-09 19:19:32.003762 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_6926c17a-76e1-49b8-a9ff-079a205d3c6b/sg-core/0.log 2026-03-09 19:19:32.003783 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-6d45c85556-w6k7b_eef0c4bd-2bde-490b-872a-eda5cac560eb/ 2026-03-09 19:19:32.003805 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-6d45c85556-w6k7b_eef0c4bd-2bde-490b-872a-eda5cac560eb/keystone-api/ 2026-03-09 19:19:32.003825 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-6d45c85556-w6k7b_eef0c4bd-2bde-490b-872a-eda5cac560eb/keystone-api/0.log 2026-03-09 19:19:32.003846 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29551381-c5bp4_98d8cd55-a4bc-446d-a770-ed57e35aeccb/ 2026-03-09 19:19:32.003867 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29551381-c5bp4_98d8cd55-a4bc-446d-a770-ed57e35aeccb/keystone-cron/ 2026-03-09 19:19:32.003888 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29551381-c5bp4_98d8cd55-a4bc-446d-a770-ed57e35aeccb/keystone-cron/0.log 2026-03-09 19:19:32.003908 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_c66df9ab-03fb-42fa-b3ef-9f3064523682/ 2026-03-09 19:19:32.003929 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_c66df9ab-03fb-42fa-b3ef-9f3064523682/kube-state-metrics/ 2026-03-09 19:19:32.003950 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_c66df9ab-03fb-42fa-b3ef-9f3064523682/kube-state-metrics/0.log 2026-03-09 19:19:32.003971 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_0277ce9b-9597-40cc-9339-51cf5dc9d98d/ 2026-03-09 19:19:32.003993 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_0277ce9b-9597-40cc-9339-51cf5dc9d98d/memcached/ 2026-03-09 19:19:32.004030 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_0277ce9b-9597-40cc-9339-51cf5dc9d98d/memcached/0.log.gz 2026-03-09 19:19:32.004068 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_e0cec899-aa83-4720-8f75-bc2fc5002a28/ 2026-03-09 19:19:32.004091 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_e0cec899-aa83-4720-8f75-bc2fc5002a28/galera/ 2026-03-09 19:19:32.004113 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_e0cec899-aa83-4720-8f75-bc2fc5002a28/galera/0.log 2026-03-09 19:19:32.004134 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_e0cec899-aa83-4720-8f75-bc2fc5002a28/mysql-bootstrap/ 2026-03-09 19:19:32.004155 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_e0cec899-aa83-4720-8f75-bc2fc5002a28/mysql-bootstrap/0.log 2026-03-09 19:19:32.004176 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_a388f45b-e428-4530-b5cf-71879e545f6e/ 2026-03-09 19:19:32.004197 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_a388f45b-e428-4530-b5cf-71879e545f6e/openstackclient/ 2026-03-09 19:19:32.004218 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_a388f45b-e428-4530-b5cf-71879e545f6e/openstackclient/0.log 2026-03-09 19:19:32.004239 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/ 2026-03-09 19:19:32.004260 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/config-reloader/ 2026-03-09 19:19:32.004281 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/config-reloader/0.log 2026-03-09 19:19:32.004302 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/init-config-reloader/ 2026-03-09 19:19:32.004323 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/init-config-reloader/0.log 2026-03-09 19:19:32.004344 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/prometheus/ 2026-03-09 19:19:32.004365 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/prometheus/0.log 2026-03-09 19:19:32.004386 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/thanos-sidecar/ 2026-03-09 19:19:32.004407 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_cd8107e4-08e5-4aea-aaf9-3a6421c9dc0f/thanos-sidecar/0.log 2026-03-09 19:19:32.004428 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_b4cf48ce-38c9-4dd4-b712-311a92dd29b6/ 2026-03-09 19:19:32.004448 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_b4cf48ce-38c9-4dd4-b712-311a92dd29b6/rabbitmq/ 2026-03-09 19:19:32.004469 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_b4cf48ce-38c9-4dd4-b712-311a92dd29b6/rabbitmq/0.log 2026-03-09 19:19:32.004495 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_b4cf48ce-38c9-4dd4-b712-311a92dd29b6/setup-container/ 2026-03-09 19:19:32.004517 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_b4cf48ce-38c9-4dd4-b712-311a92dd29b6/setup-container/0.log 2026-03-09 19:19:32.004539 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ace06b27-8092-4676-9bae-4df7c1044b98/ 2026-03-09 19:19:32.004561 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ace06b27-8092-4676-9bae-4df7c1044b98/rabbitmq/ 2026-03-09 19:19:32.004584 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ace06b27-8092-4676-9bae-4df7c1044b98/rabbitmq/0.log 2026-03-09 19:19:32.004606 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ace06b27-8092-4676-9bae-4df7c1044b98/setup-container/ 2026-03-09 19:19:32.004627 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ace06b27-8092-4676-9bae-4df7c1044b98/setup-container/0.log 2026-03-09 19:19:32.004648 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/ 2026-03-09 19:19:32.004670 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/event-filter.html 2026-03-09 19:19:32.004692 | controller | cL+++++++++ ci-framework-data/logs/openstack-must-gather/latest -> quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3 2026-03-09 19:19:32.004714 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/must-gather.logs 2026-03-09 19:19:32.004736 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/timestamp 2026-03-09 19:19:32.004757 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ 2026-03-09 19:19:32.004779 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/all-containers-cpu-top.log 2026-03-09 19:19:32.004800 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/gather.logs 2026-03-09 19:19:32.004822 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/must-gather.tar.xz 2026-03-09 19:19:32.004844 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/packagemanifests.gz 2026-03-09 19:19:32.004865 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/run.log 2026-03-09 19:19:32.004887 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/version 2026-03-09 19:19:32.004909 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/ 2026-03-09 19:19:32.004931 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.barbican.openstack.org.yaml 2026-03-09 19:19:32.004958 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.baremetal.openstack.org.yaml 2026-03-09 19:19:32.004981 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.cinder.openstack.org.yaml 2026-03-09 19:19:32.005004 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.client.openstack.org.yaml 2026-03-09 19:19:32.005050 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.core.openstack.org.yaml 2026-03-09 19:19:32.005075 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.dataplane.openstack.org.yaml 2026-03-09 19:19:32.005097 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.designate.openstack.org.yaml 2026-03-09 19:19:32.005119 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.glance.openstack.org.yaml 2026-03-09 19:19:32.005140 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.heat.openstack.org.yaml 2026-03-09 19:19:32.005162 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.horizon.openstack.org.yaml 2026-03-09 19:19:32.005184 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.instanceha.openstack.org.yaml 2026-03-09 19:19:32.005206 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.ironic.openstack.org.yaml 2026-03-09 19:19:32.005227 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.keystone.openstack.org.yaml 2026-03-09 19:19:32.005249 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.manila.openstack.org.yaml 2026-03-09 19:19:32.005271 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.mariadb.openstack.org.yaml 2026-03-09 19:19:32.005301 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.memcached.openstack.org.yaml 2026-03-09 19:19:32.005328 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.network.openstack.org.yaml 2026-03-09 19:19:32.005350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.neutron.openstack.org.yaml 2026-03-09 19:19:32.005372 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.nova.openstack.org.yaml 2026-03-09 19:19:32.005394 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.octavia.openstack.org.yaml 2026-03-09 19:19:32.005415 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.operator.openstack.org.yaml 2026-03-09 19:19:32.005437 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.ovn.openstack.org.yaml 2026-03-09 19:19:32.005459 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.placement.openstack.org.yaml 2026-03-09 19:19:32.005481 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.rabbitmq.com.yaml 2026-03-09 19:19:32.005502 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.rabbitmq.openstack.org.yaml 2026-03-09 19:19:32.005523 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.redis.openstack.org.yaml 2026-03-09 19:19:32.005545 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.swift.openstack.org.yaml 2026-03-09 19:19:32.005566 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.telemetry.openstack.org.yaml 2026-03-09 19:19:32.005588 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.test.openstack.org.yaml 2026-03-09 19:19:32.005609 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.topology.openstack.org.yaml 2026-03-09 19:19:32.005631 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.watcher.openstack.org.yaml 2026-03-09 19:19:32.005657 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ 2026-03-09 19:19:32.005679 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ansibletests.test.openstack.org.yaml 2026-03-09 19:19:32.005701 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/autoscalings.telemetry.openstack.org.yaml 2026-03-09 19:19:32.005722 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicanapis.barbican.openstack.org.yaml 2026-03-09 19:19:32.005743 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicankeystonelisteners.barbican.openstack.org.yaml 2026-03-09 19:19:32.005765 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicans.barbican.openstack.org.yaml 2026-03-09 19:19:32.005786 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicanworkers.barbican.openstack.org.yaml 2026-03-09 19:19:32.005807 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/bgpconfigurations.network.openstack.org.yaml 2026-03-09 19:19:32.005828 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ceilometers.telemetry.openstack.org.yaml 2026-03-09 19:19:32.005850 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderapis.cinder.openstack.org.yaml 2026-03-09 19:19:32.005871 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderbackups.cinder.openstack.org.yaml 2026-03-09 19:19:32.005891 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinders.cinder.openstack.org.yaml 2026-03-09 19:19:32.005912 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderschedulers.cinder.openstack.org.yaml 2026-03-09 19:19:32.005932 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cindervolumes.cinder.openstack.org.yaml 2026-03-09 19:19:32.005953 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkitties.telemetry.openstack.org.yaml 2026-03-09 19:19:32.005978 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkittyapis.telemetry.openstack.org.yaml 2026-03-09 19:19:32.006000 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkittyprocs.telemetry.openstack.org.yaml 2026-03-09 19:19:32.006045 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateapis.designate.openstack.org.yaml 2026-03-09 19:19:32.006070 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatebackendbind9s.designate.openstack.org.yaml 2026-03-09 19:19:32.006094 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatecentrals.designate.openstack.org.yaml 2026-03-09 19:19:32.006116 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatemdnses.designate.openstack.org.yaml 2026-03-09 19:19:32.006139 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateproducers.designate.openstack.org.yaml 2026-03-09 19:19:32.006161 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designates.designate.openstack.org.yaml 2026-03-09 19:19:32.006183 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateunbounds.designate.openstack.org.yaml 2026-03-09 19:19:32.006206 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateworkers.designate.openstack.org.yaml 2026-03-09 19:19:32.006228 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/dnsdata.network.openstack.org.yaml 2026-03-09 19:19:32.006250 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/dnsmasqs.network.openstack.org.yaml 2026-03-09 19:19:32.006272 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galerabackups.mariadb.openstack.org.yaml 2026-03-09 19:19:32.006294 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galerarestores.mariadb.openstack.org.yaml 2026-03-09 19:19:32.006316 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galeras.mariadb.openstack.org.yaml 2026-03-09 19:19:32.006343 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/glanceapis.glance.openstack.org.yaml 2026-03-09 19:19:32.006365 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/glances.glance.openstack.org.yaml 2026-03-09 19:19:32.006387 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatapis.heat.openstack.org.yaml 2026-03-09 19:19:32.006417 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatcfnapis.heat.openstack.org.yaml 2026-03-09 19:19:32.006439 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatengines.heat.openstack.org.yaml 2026-03-09 19:19:32.006462 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heats.heat.openstack.org.yaml 2026-03-09 19:19:32.006484 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/horizons.horizon.openstack.org.yaml 2026-03-09 19:19:32.006506 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/horizontests.test.openstack.org.yaml 2026-03-09 19:19:32.006528 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/instancehas.instanceha.openstack.org.yaml 2026-03-09 19:19:32.006550 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ipsets.network.openstack.org.yaml 2026-03-09 19:19:32.006571 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicapis.ironic.openstack.org.yaml 2026-03-09 19:19:32.006593 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicconductors.ironic.openstack.org.yaml 2026-03-09 19:19:32.006614 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicinspectors.ironic.openstack.org.yaml 2026-03-09 19:19:32.006636 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicneutronagents.ironic.openstack.org.yaml 2026-03-09 19:19:32.006657 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironics.ironic.openstack.org.yaml 2026-03-09 19:19:32.006683 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneapis.keystone.openstack.org.yaml 2026-03-09 19:19:32.006706 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneapplicationcredentials.keystone.openstack.org.yaml 2026-03-09 19:19:32.006727 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneendpoints.keystone.openstack.org.yaml 2026-03-09 19:19:32.006749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneservices.keystone.openstack.org.yaml 2026-03-09 19:19:32.006771 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/loggings.telemetry.openstack.org.yaml 2026-03-09 19:19:32.006793 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilaapis.manila.openstack.org.yaml 2026-03-09 19:19:32.006815 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilas.manila.openstack.org.yaml 2026-03-09 19:19:32.006837 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilaschedulers.manila.openstack.org.yaml 2026-03-09 19:19:32.006859 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilashares.manila.openstack.org.yaml 2026-03-09 19:19:32.006880 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/mariadbaccounts.mariadb.openstack.org.yaml 2026-03-09 19:19:32.006902 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/mariadbdatabases.mariadb.openstack.org.yaml 2026-03-09 19:19:32.006923 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/memcacheds.memcached.openstack.org.yaml 2026-03-09 19:19:32.006945 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/metricstorages.telemetry.openstack.org.yaml 2026-03-09 19:19:32.006966 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/netconfigs.network.openstack.org.yaml 2026-03-09 19:19:32.006988 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/neutronapis.neutron.openstack.org.yaml 2026-03-09 19:19:32.007009 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/nova.nova.openstack.org.yaml 2026-03-09 19:19:32.007061 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaapis.nova.openstack.org.yaml 2026-03-09 19:19:32.007085 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novacells.nova.openstack.org.yaml 2026-03-09 19:19:32.007107 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novacomputes.nova.openstack.org.yaml 2026-03-09 19:19:32.007128 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaconductors.nova.openstack.org.yaml 2026-03-09 19:19:32.007150 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novametadata.nova.openstack.org.yaml 2026-03-09 19:19:32.007171 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novanovncproxies.nova.openstack.org.yaml 2026-03-09 19:19:32.007193 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaschedulers.nova.openstack.org.yaml 2026-03-09 19:19:32.007214 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviaamphoracontrollers.octavia.openstack.org.yaml 2026-03-09 19:19:32.007236 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviaapis.octavia.openstack.org.yaml 2026-03-09 19:19:32.007258 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviarsyslogs.octavia.openstack.org.yaml 2026-03-09 19:19:32.007280 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octavias.octavia.openstack.org.yaml 2026-03-09 19:19:32.007302 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackbaremetalsets.baremetal.openstack.org.yaml 2026-03-09 19:19:32.007324 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackclients.client.openstack.org.yaml 2026-03-09 19:19:32.007346 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackcontrolplanes.core.openstack.org.yaml 2026-03-09 19:19:32.007368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplanedeployments.dataplane.openstack.org.yaml 2026-03-09 19:19:32.007407 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplanenodesets.dataplane.openstack.org.yaml 2026-03-09 19:19:32.007430 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplaneservices.dataplane.openstack.org.yaml 2026-03-09 19:19:32.007452 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackprovisionservers.baremetal.openstack.org.yaml 2026-03-09 19:19:32.007473 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstacks.operator.openstack.org.yaml 2026-03-09 19:19:32.007494 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackversions.core.openstack.org.yaml 2026-03-09 19:19:32.007515 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovncontrollers.ovn.openstack.org.yaml 2026-03-09 19:19:32.007537 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovndbclusters.ovn.openstack.org.yaml 2026-03-09 19:19:32.007575 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovnnorthds.ovn.openstack.org.yaml 2026-03-09 19:19:32.007597 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/placementapis.placement.openstack.org.yaml 2026-03-09 19:19:32.007621 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqclusters.rabbitmq.com.yaml 2026-03-09 19:19:32.007643 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqpolicies.rabbitmq.openstack.org.yaml 2026-03-09 19:19:32.007665 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqs.rabbitmq.openstack.org.yaml 2026-03-09 19:19:32.007687 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqusers.rabbitmq.openstack.org.yaml 2026-03-09 19:19:32.007709 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqvhosts.rabbitmq.openstack.org.yaml 2026-03-09 19:19:32.007731 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/redises.redis.openstack.org.yaml 2026-03-09 19:19:32.007758 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/reservations.network.openstack.org.yaml 2026-03-09 19:19:32.007781 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftproxies.swift.openstack.org.yaml 2026-03-09 19:19:32.007803 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftrings.swift.openstack.org.yaml 2026-03-09 19:19:32.007825 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swifts.swift.openstack.org.yaml 2026-03-09 19:19:32.007847 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftstorages.swift.openstack.org.yaml 2026-03-09 19:19:32.007869 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/telemetries.telemetry.openstack.org.yaml 2026-03-09 19:19:32.007890 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/tempests.test.openstack.org.yaml 2026-03-09 19:19:32.007912 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/tobikoes.test.openstack.org.yaml 2026-03-09 19:19:32.007933 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/topologies.topology.openstack.org.yaml 2026-03-09 19:19:32.007954 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/transporturls.rabbitmq.openstack.org.yaml 2026-03-09 19:19:32.007975 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherapis.watcher.openstack.org.yaml 2026-03-09 19:19:32.007996 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherappliers.watcher.openstack.org.yaml 2026-03-09 19:19:32.008044 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherdecisionengines.watcher.openstack.org.yaml 2026-03-09 19:19:32.008070 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watchers.watcher.openstack.org.yaml 2026-03-09 19:19:32.008093 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/ 2026-03-09 19:19:32.008121 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/cluster-observability-operator.v1.3.1 2026-03-09 19:19:32.008144 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/kubernetes-nmstate-operator.4.18.0-202602172216 2026-03-09 19:19:32.008167 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/metallb-operator.v4.18.0-202602140741 2026-03-09 19:19:32.008189 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/openstack-operator.v0.6.0 2026-03-09 19:19:32.008210 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/watcher-operator.v0.0.1 2026-03-09 19:19:32.008232 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/ 2026-03-09 19:19:32.008254 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/endpoints 2026-03-09 19:19:32.008275 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/services 2026-03-09 19:19:32.008296 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/ovn/ 2026-03-09 19:19:32.008317 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/rabbitmq/ 2026-03-09 19:19:32.008338 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/ 2026-03-09 19:19:32.008360 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/watcher-kuttl-default/ 2026-03-09 19:19:32.008381 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/watcher-kuttl-default/openstack-galera-0-openstack_databases.sql.gz 2026-03-09 19:19:32.008402 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/ 2026-03-09 19:19:32.008424 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/ 2026-03-09 19:19:32.008445 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/all_resources.log 2026-03-09 19:19:32.008472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/events.log 2026-03-09 19:19:32.008494 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/nad.log 2026-03-09 19:19:32.008516 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pvc.log 2026-03-09 19:19:32.008538 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/buildconfig/ 2026-03-09 19:19:32.008560 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/cert/ 2026-03-09 19:19:32.008581 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/certificaterequests/ 2026-03-09 19:19:32.008603 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/configmaps/ 2026-03-09 19:19:32.008625 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/cronjobs/ 2026-03-09 19:19:32.008647 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/daemonset/ 2026-03-09 19:19:32.008668 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/ 2026-03-09 19:19:32.008689 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager-cainjector.yaml 2026-03-09 19:19:32.008710 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager-webhook.yaml 2026-03-09 19:19:32.008732 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager.yaml 2026-03-09 19:19:32.008754 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/imagestream/ 2026-03-09 19:19:32.008775 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/installplans/ 2026-03-09 19:19:32.008796 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/issuer/ 2026-03-09 19:19:32.008822 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/jobs/ 2026-03-09 19:19:32.008844 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/poddisruptionbudgets/ 2026-03-09 19:19:32.008865 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/ 2026-03-09 19:19:32.008886 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-zmpfr/ 2026-03-09 19:19:32.008907 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-zmpfr/cert-manager-545d4d4674-zmpfr-describe 2026-03-09 19:19:32.008930 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-zmpfr/logs/ 2026-03-09 19:19:32.008952 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-zmpfr/logs/cert-manager-controller.log 2026-03-09 19:19:32.008974 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-2t9gn/ 2026-03-09 19:19:32.008996 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-2t9gn/cert-manager-cainjector-5545bd876-2t9gn-describe 2026-03-09 19:19:32.009041 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-2t9gn/logs/ 2026-03-09 19:19:32.009066 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-2t9gn/logs/cert-manager-cainjector.log 2026-03-09 19:19:32.009089 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-8k2s6/ 2026-03-09 19:19:32.009112 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-8k2s6/cert-manager-webhook-6888856db4-8k2s6-describe 2026-03-09 19:19:32.009134 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-8k2s6/logs/ 2026-03-09 19:19:32.009160 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-8k2s6/logs/cert-manager-webhook.log 2026-03-09 19:19:32.009182 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/ 2026-03-09 19:19:32.009212 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-545d4d4674.yaml 2026-03-09 19:19:32.009234 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-cainjector-5545bd876.yaml 2026-03-09 19:19:32.009256 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-webhook-6888856db4.yaml 2026-03-09 19:19:32.009278 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/routes/ 2026-03-09 19:19:32.009300 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/ 2026-03-09 19:19:32.009322 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager-cainjector.yaml 2026-03-09 19:19:32.009344 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager-webhook.yaml 2026-03-09 19:19:32.009366 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager.yaml 2026-03-09 19:19:32.009388 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/statefulset/ 2026-03-09 19:19:32.009410 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/subscriptions/ 2026-03-09 19:19:32.009431 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/ 2026-03-09 19:19:32.009452 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/all_resources.log 2026-03-09 19:19:32.009482 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/events.log 2026-03-09 19:19:32.009504 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/nad.log 2026-03-09 19:19:32.009526 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pvc.log 2026-03-09 19:19:32.009547 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/buildconfig/ 2026-03-09 19:19:32.009568 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/cert/ 2026-03-09 19:19:32.009590 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/certificaterequests/ 2026-03-09 19:19:32.009612 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/configmaps/ 2026-03-09 19:19:32.009723 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/cronjobs/ 2026-03-09 19:19:32.009757 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ 2026-03-09 19:19:32.009783 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/ 2026-03-09 19:19:32.009807 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/ctlplane.yaml 2026-03-09 19:19:32.009830 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/designateext.yaml 2026-03-09 19:19:32.009853 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/internalapi.yaml 2026-03-09 19:19:32.009875 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/storage.yaml 2026-03-09 19:19:32.009897 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/tenant.yaml 2026-03-09 19:19:32.009925 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/ 2026-03-09 19:19:32.009948 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/ctlplane.yaml 2026-03-09 19:19:32.009970 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/designateext.yaml 2026-03-09 19:19:32.009992 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/internalapi.yaml 2026-03-09 19:19:32.010031 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/storage.yaml 2026-03-09 19:19:32.010061 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/tenant.yaml 2026-03-09 19:19:32.010088 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/metallbs.metallb.io/ 2026-03-09 19:19:32.010111 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/metallbs.metallb.io/metallb.yaml 2026-03-09 19:19:32.010133 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/ 2026-03-09 19:19:32.010155 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/frr-k8s.yaml 2026-03-09 19:19:32.010177 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/speaker.yaml 2026-03-09 19:19:32.010199 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/ 2026-03-09 19:19:32.010220 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/controller.yaml 2026-03-09 19:19:32.010243 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/frr-k8s-webhook-server.yaml 2026-03-09 19:19:32.010264 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/metallb-operator-controller-manager.yaml 2026-03-09 19:19:32.010292 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/metallb-operator-webhook-server.yaml 2026-03-09 19:19:32.010315 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/imagestream/ 2026-03-09 19:19:32.010337 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/installplans/ 2026-03-09 19:19:32.010359 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/installplans/install-zbdjq 2026-03-09 19:19:32.010380 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/issuer/ 2026-03-09 19:19:32.010402 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/jobs/ 2026-03-09 19:19:32.010423 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/poddisruptionbudgets/ 2026-03-09 19:19:32.010445 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/ 2026-03-09 19:19:32.010466 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-86ddb6bd46-jfl8k/ 2026-03-09 19:19:32.010488 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-86ddb6bd46-jfl8k/controller-86ddb6bd46-jfl8k-describe 2026-03-09 19:19:32.010510 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-86ddb6bd46-jfl8k/logs/ 2026-03-09 19:19:32.010531 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-86ddb6bd46-jfl8k/logs/controller.log.gz 2026-03-09 19:19:32.010553 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-86ddb6bd46-jfl8k/logs/kube-rbac-proxy.log 2026-03-09 19:19:32.010574 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/ 2026-03-09 19:19:32.010600 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/frr-k8s-pxc5m-describe 2026-03-09 19:19:32.010622 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/ 2026-03-09 19:19:32.010643 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/controller.log 2026-03-09 19:19:32.010665 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/cp-frr-files.log 2026-03-09 19:19:32.010686 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/cp-metrics.log 2026-03-09 19:19:32.010707 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/cp-reloader.log 2026-03-09 19:19:32.010728 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/frr-metrics.log 2026-03-09 19:19:32.010749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/frr.log.gz 2026-03-09 19:19:32.010770 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/kube-rbac-proxy-frr.log 2026-03-09 19:19:32.010791 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/kube-rbac-proxy.log 2026-03-09 19:19:32.010812 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-pxc5m/logs/reloader.log 2026-03-09 19:19:32.010833 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-7f989f654f-7q47l/ 2026-03-09 19:19:32.010854 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-7f989f654f-7q47l/frr-k8s-webhook-server-7f989f654f-7q47l-describe 2026-03-09 19:19:32.010875 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-7f989f654f-7q47l/logs/ 2026-03-09 19:19:32.010902 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-7f989f654f-7q47l/logs/frr-k8s-webhook-server.log 2026-03-09 19:19:32.010924 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-858bc4f469-wp8gj/ 2026-03-09 19:19:32.010947 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-858bc4f469-wp8gj/metallb-operator-controller-manager-858bc4f469-wp8gj-describe 2026-03-09 19:19:32.010969 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-858bc4f469-wp8gj/logs/ 2026-03-09 19:19:32.010991 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-858bc4f469-wp8gj/logs/manager.log 2026-03-09 19:19:32.011032 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-5f89859c4b-c6xkg/ 2026-03-09 19:19:32.011062 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-5f89859c4b-c6xkg/metallb-operator-webhook-server-5f89859c4b-c6xkg-describe 2026-03-09 19:19:32.011085 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-5f89859c4b-c6xkg/logs/ 2026-03-09 19:19:32.011107 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-5f89859c4b-c6xkg/logs/webhook-server.log 2026-03-09 19:19:32.011129 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-5sdkw/ 2026-03-09 19:19:32.011151 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-5sdkw/speaker-5sdkw-describe 2026-03-09 19:19:32.011172 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-5sdkw/logs/ 2026-03-09 19:19:32.011195 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-5sdkw/logs/kube-rbac-proxy.log 2026-03-09 19:19:32.011221 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-5sdkw/logs/speaker.log.gz 2026-03-09 19:19:32.011243 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/ 2026-03-09 19:19:32.011265 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/controller-86ddb6bd46.yaml 2026-03-09 19:19:32.011287 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/frr-k8s-webhook-server-7f989f654f.yaml 2026-03-09 19:19:32.011310 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/metallb-operator-controller-manager-858bc4f469.yaml 2026-03-09 19:19:32.011408 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/metallb-operator-webhook-server-5f89859c4b.yaml 2026-03-09 19:19:32.011449 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/routes/ 2026-03-09 19:19:32.011475 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/ 2026-03-09 19:19:32.011498 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/controller-monitor-service.yaml 2026-03-09 19:19:32.011521 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/frr-k8s-monitor-service.yaml 2026-03-09 19:19:32.011562 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/frr-k8s-webhook-service.yaml 2026-03-09 19:19:32.011599 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-controller-manager-service.yaml 2026-03-09 19:19:32.011631 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-webhook-server-service.yaml 2026-03-09 19:19:32.011657 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-webhook-service.yaml 2026-03-09 19:19:32.011685 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/speaker-monitor-service.yaml 2026-03-09 19:19:32.011708 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/webhook-service.yaml 2026-03-09 19:19:32.011730 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/statefulset/ 2026-03-09 19:19:32.011753 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/subscriptions/ 2026-03-09 19:19:32.011775 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/subscriptions/metallb-operator-sub 2026-03-09 19:19:32.011797 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/ 2026-03-09 19:19:32.011819 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/all_resources.log 2026-03-09 19:19:32.011841 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/events.log 2026-03-09 19:19:32.011863 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/nad.log 2026-03-09 19:19:32.011884 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pvc.log 2026-03-09 19:19:32.011905 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/buildconfig/ 2026-03-09 19:19:32.011927 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/cert/ 2026-03-09 19:19:32.011948 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/certificaterequests/ 2026-03-09 19:19:32.011969 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/configmaps/ 2026-03-09 19:19:32.011991 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/cronjobs/ 2026-03-09 19:19:32.012039 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/daemonset/ 2026-03-09 19:19:32.012070 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/ 2026-03-09 19:19:32.012097 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml 2026-03-09 19:19:32.012119 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/machine-api-operator.yaml 2026-03-09 19:19:32.012140 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/imagestream/ 2026-03-09 19:19:32.012162 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/installplans/ 2026-03-09 19:19:32.012183 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/issuer/ 2026-03-09 19:19:32.012205 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/jobs/ 2026-03-09 19:19:32.012226 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/poddisruptionbudgets/ 2026-03-09 19:19:32.012248 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/ 2026-03-09 19:19:32.012269 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-dbk5m/ 2026-03-09 19:19:32.012290 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-dbk5m/control-plane-machine-set-operator-78cbb6b69f-dbk5m-describe 2026-03-09 19:19:32.012312 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-dbk5m/logs/ 2026-03-09 19:19:32.012334 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-dbk5m/logs/control-plane-machine-set-operator.log 2026-03-09 19:19:32.012362 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-h8j2t/ 2026-03-09 19:19:32.012385 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-h8j2t/machine-api-operator-5694c8668f-h8j2t-describe 2026-03-09 19:19:32.012407 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-h8j2t/logs/ 2026-03-09 19:19:32.012429 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-h8j2t/logs/kube-rbac-proxy.log 2026-03-09 19:19:32.012451 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-h8j2t/logs/machine-api-operator.log 2026-03-09 19:19:32.012473 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/ 2026-03-09 19:19:32.012494 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-78cbb6b69f.yaml 2026-03-09 19:19:32.012516 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/machine-api-operator-5694c8668f.yaml 2026-03-09 19:19:32.012537 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/routes/ 2026-03-09 19:19:32.012559 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/ 2026-03-09 19:19:32.012580 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml 2026-03-09 19:19:32.012602 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml 2026-03-09 19:19:32.012624 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-controllers.yaml 2026-03-09 19:19:32.012647 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml 2026-03-09 19:19:32.012673 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml 2026-03-09 19:19:32.012696 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator.yaml 2026-03-09 19:19:32.012718 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/statefulset/ 2026-03-09 19:19:32.012741 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/subscriptions/ 2026-03-09 19:19:32.012762 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/ 2026-03-09 19:19:32.012784 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/all_resources.log 2026-03-09 19:19:32.012805 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/events.log 2026-03-09 19:19:32.012826 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/nad.log 2026-03-09 19:19:32.012848 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pvc.log 2026-03-09 19:19:32.012869 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/buildconfig/ 2026-03-09 19:19:32.012890 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/cert/ 2026-03-09 19:19:32.012911 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/certificaterequests/ 2026-03-09 19:19:32.012932 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/configmaps/ 2026-03-09 19:19:32.012953 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/cronjobs/ 2026-03-09 19:19:32.012979 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/daemonset/ 2026-03-09 19:19:32.013002 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/deployments/ 2026-03-09 19:19:32.013051 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/deployments/marketplace-operator.yaml 2026-03-09 19:19:32.013076 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/imagestream/ 2026-03-09 19:19:32.013099 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/installplans/ 2026-03-09 19:19:32.013121 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/issuer/ 2026-03-09 19:19:32.013142 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/ 2026-03-09 19:19:32.013164 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a824662b.yaml 2026-03-09 19:19:32.013185 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c.yaml 2026-03-09 19:19:32.013207 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f081954b.yaml 2026-03-09 19:19:32.013229 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f421346.yaml 2026-03-09 19:19:32.013250 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/poddisruptionbudgets/ 2026-03-09 19:19:32.013272 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/ 2026-03-09 19:19:32.013294 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq/ 2026-03-09 19:19:32.013321 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq/0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq-describe 2026-03-09 19:19:32.013349 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq/logs/ 2026-03-09 19:19:32.013371 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq/logs/extract.log 2026-03-09 19:19:32.013392 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq/logs/pull.log 2026-03-09 19:19:32.013414 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/0e94e7566f739476ccec6d16e58de3f1c434cfa3060893f90f3e473a82d7bqq/logs/util.log 2026-03-09 19:19:32.013435 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4/ 2026-03-09 19:19:32.013457 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4-describe 2026-03-09 19:19:32.013478 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4/logs/ 2026-03-09 19:19:32.013499 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4/logs/extract.log 2026-03-09 19:19:32.013520 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4/logs/pull.log 2026-03-09 19:19:32.013541 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5rfnx4/logs/util.log 2026-03-09 19:19:32.013563 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6/ 2026-03-09 19:19:32.013588 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6-describe 2026-03-09 19:19:32.013610 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6/logs/ 2026-03-09 19:19:32.013632 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6/logs/extract.log 2026-03-09 19:19:32.013653 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6/logs/pull.log 2026-03-09 19:19:32.013675 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f087dlv6/logs/util.log 2026-03-09 19:19:32.013696 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vhn27/ 2026-03-09 19:19:32.013718 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vhn27/certified-operators-vhn27-describe 2026-03-09 19:19:32.013739 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vhn27/logs/ 2026-03-09 19:19:32.013760 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vhn27/logs/extract-content.log 2026-03-09 19:19:32.013783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vhn27/logs/extract-utilities.log 2026-03-09 19:19:32.013810 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vhn27/logs/registry-server.log.gz 2026-03-09 19:19:32.013832 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-29tmk/ 2026-03-09 19:19:32.013854 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-29tmk/community-operators-29tmk-describe 2026-03-09 19:19:32.013880 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-29tmk/logs/ 2026-03-09 19:19:32.013901 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-29tmk/logs/extract-content.log 2026-03-09 19:19:32.013923 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-29tmk/logs/extract-utilities.log 2026-03-09 19:19:32.013945 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-29tmk/logs/registry-server.log.gz 2026-03-09 19:19:32.013966 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j/ 2026-03-09 19:19:32.013988 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j/d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j-describe 2026-03-09 19:19:32.014009 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j/logs/ 2026-03-09 19:19:32.014058 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j/logs/extract.log 2026-03-09 19:19:32.014084 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j/logs/pull.log 2026-03-09 19:19:32.014108 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/d146760600e43041070ad4572d9c23f31a62e3aefc01a54998863bc5f4b575j/logs/util.log 2026-03-09 19:19:32.014130 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-556c4/ 2026-03-09 19:19:32.014151 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-556c4/marketplace-operator-79b997595-556c4-describe 2026-03-09 19:19:32.014178 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-556c4/logs/ 2026-03-09 19:19:32.014200 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-556c4/logs/marketplace-operator.log 2026-03-09 19:19:32.014222 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-pchlh/ 2026-03-09 19:19:32.014244 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-pchlh/redhat-marketplace-pchlh-describe 2026-03-09 19:19:32.014266 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-pchlh/logs/ 2026-03-09 19:19:32.014287 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-pchlh/logs/extract-content.log 2026-03-09 19:19:32.014309 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-pchlh/logs/extract-utilities.log 2026-03-09 19:19:32.014331 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-pchlh/logs/registry-server.log 2026-03-09 19:19:32.014352 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-tc2f5/ 2026-03-09 19:19:32.014374 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-tc2f5/redhat-operators-tc2f5-describe 2026-03-09 19:19:32.014396 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-tc2f5/logs/ 2026-03-09 19:19:32.014417 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-tc2f5/logs/extract-content.log 2026-03-09 19:19:32.014438 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-tc2f5/logs/extract-utilities.log 2026-03-09 19:19:32.014468 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-tc2f5/logs/registry-server.log.gz 2026-03-09 19:19:32.014490 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/replicaset/ 2026-03-09 19:19:32.014512 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/replicaset/marketplace-operator-79b997595.yaml 2026-03-09 19:19:32.014533 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/routes/ 2026-03-09 19:19:32.014555 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/ 2026-03-09 19:19:32.014577 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/certified-operators.yaml 2026-03-09 19:19:32.014599 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/community-operators.yaml 2026-03-09 19:19:32.014621 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/marketplace-operator-metrics.yaml 2026-03-09 19:19:32.014643 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/redhat-marketplace.yaml 2026-03-09 19:19:32.014668 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/redhat-operators.yaml 2026-03-09 19:19:32.014691 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/statefulset/ 2026-03-09 19:19:32.014712 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/subscriptions/ 2026-03-09 19:19:32.014733 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/ 2026-03-09 19:19:32.014755 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/all_resources.log 2026-03-09 19:19:32.014776 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/events.log 2026-03-09 19:19:32.014802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/nad.log 2026-03-09 19:19:32.014824 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pvc.log 2026-03-09 19:19:32.014845 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/buildconfig/ 2026-03-09 19:19:32.014866 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/cert/ 2026-03-09 19:19:32.014887 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/certificaterequests/ 2026-03-09 19:19:32.014908 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/configmaps/ 2026-03-09 19:19:32.014929 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/cronjobs/ 2026-03-09 19:19:32.014950 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/daemonset/ 2026-03-09 19:19:32.014971 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/daemonset/nmstate-handler.yaml 2026-03-09 19:19:32.014992 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/ 2026-03-09 19:19:32.015013 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-console-plugin.yaml 2026-03-09 19:19:32.015035 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-metrics.yaml 2026-03-09 19:19:32.015056 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-operator.yaml 2026-03-09 19:19:32.015078 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-webhook.yaml 2026-03-09 19:19:32.015123 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/imagestream/ 2026-03-09 19:19:32.015151 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/installplans/ 2026-03-09 19:19:32.015173 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/installplans/install-psr92 2026-03-09 19:19:32.015195 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/issuer/ 2026-03-09 19:19:32.015217 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/jobs/ 2026-03-09 19:19:32.015238 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/poddisruptionbudgets/ 2026-03-09 19:19:32.015259 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/poddisruptionbudgets/nmstate-webhook.yaml 2026-03-09 19:19:32.015280 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/ 2026-03-09 19:19:32.015301 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-5dcbbd79cf-sk2qd/ 2026-03-09 19:19:32.015323 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-5dcbbd79cf-sk2qd/nmstate-console-plugin-5dcbbd79cf-sk2qd-describe 2026-03-09 19:19:32.015344 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-5dcbbd79cf-sk2qd/logs/ 2026-03-09 19:19:32.015366 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-5dcbbd79cf-sk2qd/logs/nmstate-console-plugin.log 2026-03-09 19:19:32.015387 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-msftq/ 2026-03-09 19:19:32.015409 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-msftq/nmstate-handler-msftq-describe 2026-03-09 19:19:32.015430 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-msftq/logs/ 2026-03-09 19:19:32.015456 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-msftq/logs/nmstate-handler.log 2026-03-09 19:19:32.015478 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-69594cc75-g2mff/ 2026-03-09 19:19:32.015500 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-69594cc75-g2mff/nmstate-metrics-69594cc75-g2mff-describe 2026-03-09 19:19:32.015520 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-69594cc75-g2mff/logs/ 2026-03-09 19:19:32.015556 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-69594cc75-g2mff/logs/kube-rbac-proxy.log 2026-03-09 19:19:32.015580 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-69594cc75-g2mff/logs/nmstate-metrics.log 2026-03-09 19:19:32.015601 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-75c5dccd6c-2j8gv/ 2026-03-09 19:19:32.015622 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-75c5dccd6c-2j8gv/nmstate-operator-75c5dccd6c-2j8gv-describe 2026-03-09 19:19:32.015644 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-75c5dccd6c-2j8gv/logs/ 2026-03-09 19:19:32.015669 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-75c5dccd6c-2j8gv/logs/nmstate-operator.log 2026-03-09 19:19:32.015691 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-786f45cff4-hnr59/ 2026-03-09 19:19:32.015713 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-786f45cff4-hnr59/nmstate-webhook-786f45cff4-hnr59-describe 2026-03-09 19:19:32.015734 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-786f45cff4-hnr59/logs/ 2026-03-09 19:19:32.015761 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-786f45cff4-hnr59/logs/nmstate-webhook.log 2026-03-09 19:19:32.015783 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/ 2026-03-09 19:19:32.015805 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-console-plugin-5dcbbd79cf.yaml 2026-03-09 19:19:32.015827 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-metrics-69594cc75.yaml 2026-03-09 19:19:32.015849 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-operator-75c5dccd6c.yaml 2026-03-09 19:19:32.015871 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-webhook-786f45cff4.yaml 2026-03-09 19:19:32.015892 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/routes/ 2026-03-09 19:19:32.015913 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/ 2026-03-09 19:19:32.015935 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-console-plugin.yaml 2026-03-09 19:19:32.015956 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-monitor.yaml 2026-03-09 19:19:32.015977 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-webhook.yaml 2026-03-09 19:19:32.015999 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/statefulset/ 2026-03-09 19:19:32.016043 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/subscriptions/ 2026-03-09 19:19:32.016068 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/subscriptions/kubernetes-nmstate-operator 2026-03-09 19:19:32.016090 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/ 2026-03-09 19:19:32.016117 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/all_resources.log 2026-03-09 19:19:32.016139 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/events.log 2026-03-09 19:19:32.016161 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/nad.log 2026-03-09 19:19:32.016184 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pvc.log 2026-03-09 19:19:32.016206 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/buildconfig/ 2026-03-09 19:19:32.016227 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/cert/ 2026-03-09 19:19:32.016249 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/certificaterequests/ 2026-03-09 19:19:32.016270 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/configmaps/ 2026-03-09 19:19:32.016292 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/cronjobs/ 2026-03-09 19:19:32.016314 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/daemonset/ 2026-03-09 19:19:32.016335 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/ 2026-03-09 19:19:32.016357 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/obo-prometheus-operator-admission-webhook.yaml 2026-03-09 19:19:32.016379 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/obo-prometheus-operator.yaml 2026-03-09 19:19:32.016400 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/observability-operator.yaml 2026-03-09 19:19:32.016422 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/observability-ui-dashboards.yaml 2026-03-09 19:19:32.016448 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/perses-operator.yaml 2026-03-09 19:19:32.016470 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/imagestream/ 2026-03-09 19:19:32.016492 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/installplans/ 2026-03-09 19:19:32.016513 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/installplans/install-8ltlk 2026-03-09 19:19:32.016534 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/issuer/ 2026-03-09 19:19:32.016555 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/jobs/ 2026-03-09 19:19:32.016576 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/poddisruptionbudgets/ 2026-03-09 19:19:32.016597 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/poddisruptionbudgets/obo-prometheus-operator-admission-webhook.yaml 2026-03-09 19:19:32.016618 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/ 2026-03-09 19:19:32.016639 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-cqfq5/ 2026-03-09 19:19:32.016660 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-cqfq5/obo-prometheus-operator-68bc856cb9-cqfq5-describe 2026-03-09 19:19:32.016681 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-cqfq5/logs/ 2026-03-09 19:19:32.016702 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-cqfq5/logs/prometheus-operator.log 2026-03-09 19:19:32.016724 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-559887c586-44lfv/ 2026-03-09 19:19:32.016750 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-559887c586-44lfv/obo-prometheus-operator-admission-webhook-559887c586-44lfv-describe 2026-03-09 19:19:32.016772 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-559887c586-44lfv/logs/ 2026-03-09 19:19:32.016793 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-559887c586-44lfv/logs/prometheus-operator-admission-webhook.log 2026-03-09 19:19:32.016814 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-559887c586-xfmsd/ 2026-03-09 19:19:32.016836 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-559887c586-xfmsd/obo-prometheus-operator-admission-webhook-559887c586-xfmsd-describe 2026-03-09 19:19:32.016857 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-559887c586-xfmsd/logs/ 2026-03-09 19:19:32.016878 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-559887c586-xfmsd/logs/prometheus-operator-admission-webhook.log 2026-03-09 19:19:32.016900 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-j8xx6/ 2026-03-09 19:19:32.016921 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-j8xx6/observability-operator-59bdc8b94-j8xx6-describe 2026-03-09 19:19:32.016943 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-j8xx6/logs/ 2026-03-09 19:19:32.016964 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-j8xx6/logs/operator.log 2026-03-09 19:19:32.016986 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-hkkkk/ 2026-03-09 19:19:32.017011 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-hkkkk/observability-ui-dashboards-66cbf594b5-hkkkk-describe 2026-03-09 19:19:32.017060 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-hkkkk/logs/ 2026-03-09 19:19:32.017084 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-hkkkk/logs/observability-ui-dashboards.log 2026-03-09 19:19:32.017106 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-p679h/ 2026-03-09 19:19:32.017129 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-p679h/perses-operator-5bf474d74f-p679h-describe 2026-03-09 19:19:32.017151 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-p679h/logs/ 2026-03-09 19:19:32.017172 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-p679h/logs/perses-operator.log 2026-03-09 19:19:32.017194 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/ 2026-03-09 19:19:32.017216 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/obo-prometheus-operator-68bc856cb9.yaml 2026-03-09 19:19:32.017238 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/obo-prometheus-operator-admission-webhook-559887c586.yaml 2026-03-09 19:19:32.017260 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/observability-operator-59bdc8b94.yaml 2026-03-09 19:19:32.017282 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/observability-ui-dashboards-66cbf594b5.yaml 2026-03-09 19:19:32.017304 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/perses-operator-5bf474d74f.yaml 2026-03-09 19:19:32.017331 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/routes/ 2026-03-09 19:19:32.017354 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/ 2026-03-09 19:19:32.017376 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator-admission-webhook-service.yaml 2026-03-09 19:19:32.017398 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator-admission-webhook.yaml 2026-03-09 19:19:32.017420 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator.yaml 2026-03-09 19:19:32.017442 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/observability-operator.yaml 2026-03-09 19:19:32.017464 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/observability-ui-dashboards.yaml 2026-03-09 19:19:32.017485 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/statefulset/ 2026-03-09 19:19:32.017506 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/ 2026-03-09 19:19:32.017528 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/cluster-observability-operator 2026-03-09 19:19:32.017549 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/observability-operator 2026-03-09 19:19:32.017570 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/ 2026-03-09 19:19:32.017592 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/all_resources.log 2026-03-09 19:19:32.017613 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/events.log 2026-03-09 19:19:32.017639 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/nad.log 2026-03-09 19:19:32.017661 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pvc.log 2026-03-09 19:19:32.017683 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/buildconfig/ 2026-03-09 19:19:32.017705 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/ 2026-03-09 19:19:32.017728 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/barbican-operator-metrics-certs.yaml 2026-03-09 19:19:32.017750 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/cinder-operator-metrics-certs.yaml 2026-03-09 19:19:32.017772 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/designate-operator-metrics-certs.yaml 2026-03-09 19:19:32.017794 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/glance-operator-metrics-certs.yaml 2026-03-09 19:19:32.017816 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/heat-operator-metrics-certs.yaml 2026-03-09 19:19:32.017837 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/horizon-operator-metrics-certs.yaml 2026-03-09 19:19:32.017858 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/infra-operator-metrics-certs.yaml 2026-03-09 19:19:32.017879 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/infra-operator-serving-cert.yaml 2026-03-09 19:19:32.017900 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/ironic-operator-metrics-certs.yaml 2026-03-09 19:19:32.017921 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/keystone-operator-metrics-certs.yaml 2026-03-09 19:19:32.017946 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/manila-operator-metrics-certs.yaml 2026-03-09 19:19:32.017968 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/mariadb-operator-metrics-certs.yaml 2026-03-09 19:19:32.017989 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/neutron-operator-metrics-certs.yaml 2026-03-09 19:19:32.018010 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/nova-operator-metrics-certs.yaml 2026-03-09 19:19:32.018057 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/octavia-operator-metrics-certs.yaml 2026-03-09 19:19:32.018083 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-baremetal-operator-metrics-certs.yaml 2026-03-09 19:19:32.018106 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-baremetal-operator-serving-cert.yaml 2026-03-09 19:19:32.018129 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-operator-metrics-certs.yaml 2026-03-09 19:19:32.018154 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-operator-serving-cert.yaml 2026-03-09 19:19:32.018175 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/ovn-operator-metrics-certs.yaml 2026-03-09 19:19:32.018197 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/placement-operator-metrics-certs.yaml 2026-03-09 19:19:32.018218 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/swift-operator-metrics-certs.yaml 2026-03-09 19:19:32.018239 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/telemetry-operator-metrics-certs.yaml 2026-03-09 19:19:32.018260 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/test-operator-metrics-certs.yaml 2026-03-09 19:19:32.018286 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/watcher-operator-metrics-certs.yaml 2026-03-09 19:19:32.018308 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/ 2026-03-09 19:19:32.018329 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/barbican-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/cinder-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018372 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/designate-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018392 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/glance-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018413 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/heat-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018434 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/horizon-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018456 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/infra-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018477 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/infra-operator-serving-cert-1.yaml 2026-03-09 19:19:32.018498 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/ironic-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018519 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/keystone-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018540 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/manila-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018565 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/mariadb-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018587 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/neutron-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018608 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/nova-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018629 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/octavia-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018651 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-baremetal-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018673 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-baremetal-operator-serving-cert-1.yaml 2026-03-09 19:19:32.018695 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018716 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-operator-serving-cert-1.yaml 2026-03-09 19:19:32.018738 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/ovn-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018758 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/placement-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018779 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/swift-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018800 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/telemetry-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018822 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/test-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018847 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/watcher-operator-metrics-certs-1.yaml 2026-03-09 19:19:32.018869 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/configmaps/ 2026-03-09 19:19:32.018890 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cronjobs/ 2026-03-09 19:19:32.018913 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/ 2026-03-09 19:19:32.018934 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/openstacks.operator.openstack.org/ 2026-03-09 19:19:32.018956 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/openstacks.operator.openstack.org/openstack.yaml 2026-03-09 19:19:32.018978 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/daemonset/ 2026-03-09 19:19:32.019000 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/ 2026-03-09 19:19:32.019046 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/barbican-operator-controller-manager.yaml 2026-03-09 19:19:32.019071 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/cinder-operator-controller-manager.yaml 2026-03-09 19:19:32.019092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/designate-operator-controller-manager.yaml 2026-03-09 19:19:32.019114 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/glance-operator-controller-manager.yaml 2026-03-09 19:19:32.019136 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/heat-operator-controller-manager.yaml 2026-03-09 19:19:32.019157 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/horizon-operator-controller-manager.yaml 2026-03-09 19:19:32.019186 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/infra-operator-controller-manager.yaml 2026-03-09 19:19:32.019208 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/ironic-operator-controller-manager.yaml 2026-03-09 19:19:32.019229 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/keystone-operator-controller-manager.yaml 2026-03-09 19:19:32.019250 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/manila-operator-controller-manager.yaml 2026-03-09 19:19:32.019271 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/mariadb-operator-controller-manager.yaml 2026-03-09 19:19:32.019293 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/neutron-operator-controller-manager.yaml 2026-03-09 19:19:32.019315 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/nova-operator-controller-manager.yaml 2026-03-09 19:19:32.019336 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/octavia-operator-controller-manager.yaml 2026-03-09 19:19:32.019358 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-baremetal-operator-controller-manager.yaml 2026-03-09 19:19:32.019379 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-operator-controller-init.yaml 2026-03-09 19:19:32.019401 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-operator-controller-manager.yaml 2026-03-09 19:19:32.019422 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/ovn-operator-controller-manager.yaml 2026-03-09 19:19:32.019444 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/placement-operator-controller-manager.yaml 2026-03-09 19:19:32.019465 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/rabbitmq-cluster-operator-manager.yaml 2026-03-09 19:19:32.019491 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/swift-operator-controller-manager.yaml 2026-03-09 19:19:32.019513 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/telemetry-operator-controller-manager.yaml 2026-03-09 19:19:32.019548 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/test-operator-controller-manager.yaml 2026-03-09 19:19:32.019571 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/watcher-operator-controller-manager.yaml 2026-03-09 19:19:32.019592 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/imagestream/ 2026-03-09 19:19:32.019614 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/ 2026-03-09 19:19:32.019636 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/install-cdsmk 2026-03-09 19:19:32.019658 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/install-qcfws 2026-03-09 19:19:32.019679 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/ 2026-03-09 19:19:32.019701 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/barbican-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.019723 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/cinder-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.019745 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/designate-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.019854 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/glance-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.019894 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/heat-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.019919 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/horizon-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.019943 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/infra-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.019966 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/ironic-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.019988 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/keystone-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020010 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/manila-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020058 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/mariadb-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020082 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/neutron-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020102 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/nova-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020118 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/octavia-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020133 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/openstack-baremetal-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020149 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/openstack-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020164 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/ovn-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020180 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/placement-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020199 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/rabbitmq-cluster-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020216 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/swift-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020232 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/telemetry-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020247 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/test-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020263 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/watcher-operator-selfsigned-issuer.yaml 2026-03-09 19:19:32.020278 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/ 2026-03-09 19:19:32.020294 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bf86164.yaml 2026-03-09 19:19:32.020310 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659c3480.yaml 2026-03-09 19:19:32.020325 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/poddisruptionbudgets/ 2026-03-09 19:19:32.020341 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ 2026-03-09 19:19:32.020356 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm/ 2026-03-09 19:19:32.020372 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm/5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm-describe 2026-03-09 19:19:32.020387 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm/logs/ 2026-03-09 19:19:32.020405 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm/logs/extract.log 2026-03-09 19:19:32.020422 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm/logs/pull.log 2026-03-09 19:19:32.020438 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/5eea759a77a44b7d379d7a90e28614a746a8848e17a3c9b1bbf53168bfgr2wm/logs/util.log 2026-03-09 19:19:32.020454 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx/ 2026-03-09 19:19:32.020470 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx/76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx-describe 2026-03-09 19:19:32.020485 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx/logs/ 2026-03-09 19:19:32.020500 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx/logs/extract.log 2026-03-09 19:19:32.020516 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx/logs/pull.log 2026-03-09 19:19:32.020531 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/76ea107b81a63add7970cc5182c9b5e3e7d7b3003be3f19b2a6bc21659g9hwx/logs/util.log 2026-03-09 19:19:32.020546 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-6db6876945-9vg4l/ 2026-03-09 19:19:32.020561 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-6db6876945-9vg4l/barbican-operator-controller-manager-6db6876945-9vg4l-describe 2026-03-09 19:19:32.020577 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-6db6876945-9vg4l/logs/ 2026-03-09 19:19:32.020596 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-6db6876945-9vg4l/logs/manager.log.gz 2026-03-09 19:19:32.020611 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-55d77d7b5c-cjvgb/ 2026-03-09 19:19:32.020626 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-55d77d7b5c-cjvgb/cinder-operator-controller-manager-55d77d7b5c-cjvgb-describe 2026-03-09 19:19:32.020641 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-55d77d7b5c-cjvgb/logs/ 2026-03-09 19:19:32.020657 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-55d77d7b5c-cjvgb/logs/manager.log.gz 2026-03-09 19:19:32.020672 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-5d87c9d997-5rbnb/ 2026-03-09 19:19:32.020688 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-5d87c9d997-5rbnb/designate-operator-controller-manager-5d87c9d997-5rbnb-describe 2026-03-09 19:19:32.020705 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-5d87c9d997-5rbnb/logs/ 2026-03-09 19:19:32.020720 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-5d87c9d997-5rbnb/logs/manager.log 2026-03-09 19:19:32.020737 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-64db6967f8-hjztj/ 2026-03-09 19:19:32.020753 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-64db6967f8-hjztj/glance-operator-controller-manager-64db6967f8-hjztj-describe 2026-03-09 19:19:32.020769 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-64db6967f8-hjztj/logs/ 2026-03-09 19:19:32.020788 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-64db6967f8-hjztj/logs/manager.log 2026-03-09 19:19:32.020804 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-cf99c678f-k4q8b/ 2026-03-09 19:19:32.020820 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-cf99c678f-k4q8b/heat-operator-controller-manager-cf99c678f-k4q8b-describe 2026-03-09 19:19:32.020836 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-cf99c678f-k4q8b/logs/ 2026-03-09 19:19:32.020851 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-cf99c678f-k4q8b/logs/manager.log 2026-03-09 19:19:32.020867 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-78bc7f9bd9-vhljc/ 2026-03-09 19:19:32.020883 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-78bc7f9bd9-vhljc/horizon-operator-controller-manager-78bc7f9bd9-vhljc-describe 2026-03-09 19:19:32.020898 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-78bc7f9bd9-vhljc/logs/ 2026-03-09 19:19:32.020914 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-78bc7f9bd9-vhljc/logs/manager.log 2026-03-09 19:19:32.020929 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-f7fcc58b9-rldv2/ 2026-03-09 19:19:32.020944 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-f7fcc58b9-rldv2/infra-operator-controller-manager-f7fcc58b9-rldv2-describe 2026-03-09 19:19:32.020959 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-f7fcc58b9-rldv2/logs/ 2026-03-09 19:19:32.020975 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-f7fcc58b9-rldv2/logs/manager.log 2026-03-09 19:19:32.020994 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-545456dc4-wzvf8/ 2026-03-09 19:19:32.021009 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-545456dc4-wzvf8/ironic-operator-controller-manager-545456dc4-wzvf8-describe 2026-03-09 19:19:32.021044 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-545456dc4-wzvf8/logs/ 2026-03-09 19:19:32.021061 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-545456dc4-wzvf8/logs/manager.log 2026-03-09 19:19:32.021079 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-7c789f89c6-pc4fs/ 2026-03-09 19:19:32.021097 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-7c789f89c6-pc4fs/keystone-operator-controller-manager-7c789f89c6-pc4fs-describe 2026-03-09 19:19:32.021113 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-7c789f89c6-pc4fs/logs/ 2026-03-09 19:19:32.021128 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-7c789f89c6-pc4fs/logs/manager.log 2026-03-09 19:19:32.021150 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-67d996989d-dhq9j/ 2026-03-09 19:19:32.021166 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-67d996989d-dhq9j/manila-operator-controller-manager-67d996989d-dhq9j-describe 2026-03-09 19:19:32.021181 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-67d996989d-dhq9j/logs/ 2026-03-09 19:19:32.021196 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-67d996989d-dhq9j/logs/manager.log 2026-03-09 19:19:32.021216 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-7b6bfb6475-9jwph/ 2026-03-09 19:19:32.021232 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-7b6bfb6475-9jwph/mariadb-operator-controller-manager-7b6bfb6475-9jwph-describe 2026-03-09 19:19:32.021247 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-7b6bfb6475-9jwph/logs/ 2026-03-09 19:19:32.021262 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-7b6bfb6475-9jwph/logs/manager.log 2026-03-09 19:19:32.021278 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-54688575f-974k8/ 2026-03-09 19:19:32.021293 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-54688575f-974k8/neutron-operator-controller-manager-54688575f-974k8-describe 2026-03-09 19:19:32.021308 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-54688575f-974k8/logs/ 2026-03-09 19:19:32.021323 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-54688575f-974k8/logs/manager.log 2026-03-09 19:19:32.021338 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-74b6b5dc96-qz976/ 2026-03-09 19:19:32.021353 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-74b6b5dc96-qz976/nova-operator-controller-manager-74b6b5dc96-qz976-describe 2026-03-09 19:19:32.021369 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-74b6b5dc96-qz976/logs/ 2026-03-09 19:19:32.021384 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-74b6b5dc96-qz976/logs/manager.log 2026-03-09 19:19:32.021403 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5d86c7ddb7-6864w/ 2026-03-09 19:19:32.021418 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5d86c7ddb7-6864w/octavia-operator-controller-manager-5d86c7ddb7-6864w-describe 2026-03-09 19:19:32.021433 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5d86c7ddb7-6864w/logs/ 2026-03-09 19:19:32.021508 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5d86c7ddb7-6864w/logs/manager.log 2026-03-09 19:19:32.021534 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-7c6767dc9cp28hq/ 2026-03-09 19:19:32.021553 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-7c6767dc9cp28hq/openstack-baremetal-operator-controller-manager-7c6767dc9cp28hq-describe 2026-03-09 19:19:32.021570 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-7c6767dc9cp28hq/logs/ 2026-03-09 19:19:32.021586 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-7c6767dc9cp28hq/logs/manager.log 2026-03-09 19:19:32.021602 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-64797568c9-7qbhc/ 2026-03-09 19:19:32.021619 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-64797568c9-7qbhc/openstack-operator-controller-manager-64797568c9-7qbhc-describe 2026-03-09 19:19:32.021636 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-64797568c9-7qbhc/logs/ 2026-03-09 19:19:32.021652 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-64797568c9-7qbhc/logs/manager.log.gz 2026-03-09 19:19:32.021673 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-cjjbr/ 2026-03-09 19:19:32.021689 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-cjjbr/openstack-operator-index-cjjbr-describe 2026-03-09 19:19:32.021705 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-cjjbr/logs/ 2026-03-09 19:19:32.021721 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-cjjbr/logs/registry-server.log 2026-03-09 19:19:32.021737 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-75684d597f-w9h2t/ 2026-03-09 19:19:32.021753 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-75684d597f-w9h2t/ovn-operator-controller-manager-75684d597f-w9h2t-describe 2026-03-09 19:19:32.021768 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-75684d597f-w9h2t/logs/ 2026-03-09 19:19:32.021783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-75684d597f-w9h2t/logs/manager.log 2026-03-09 19:19:32.021799 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-648564c9fc-wbjvw/ 2026-03-09 19:19:32.021814 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-648564c9fc-wbjvw/placement-operator-controller-manager-648564c9fc-wbjvw-describe 2026-03-09 19:19:32.021829 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-648564c9fc-wbjvw/logs/ 2026-03-09 19:19:32.021844 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-648564c9fc-wbjvw/logs/manager.log 2026-03-09 19:19:32.021860 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-xzdll/ 2026-03-09 19:19:32.021879 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-xzdll/rabbitmq-cluster-operator-manager-668c99d594-xzdll-describe 2026-03-09 19:19:32.021896 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-xzdll/logs/ 2026-03-09 19:19:32.021911 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-xzdll/logs/operator.log 2026-03-09 19:19:32.021927 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-9b9ff9f4d-k9s84/ 2026-03-09 19:19:32.021943 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-9b9ff9f4d-k9s84/swift-operator-controller-manager-9b9ff9f4d-k9s84-describe 2026-03-09 19:19:32.021958 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-9b9ff9f4d-k9s84/logs/ 2026-03-09 19:19:32.021973 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-9b9ff9f4d-k9s84/logs/manager.log 2026-03-09 19:19:32.021988 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-5fdb694969-zffr5/ 2026-03-09 19:19:32.022004 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-5fdb694969-zffr5/telemetry-operator-controller-manager-5fdb694969-zffr5-describe 2026-03-09 19:19:32.022084 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-5fdb694969-zffr5/logs/ 2026-03-09 19:19:32.022103 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-5fdb694969-zffr5/logs/manager.log.gz 2026-03-09 19:19:32.022119 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-55b5ff4dbb-z5d4k/ 2026-03-09 19:19:32.022138 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-55b5ff4dbb-z5d4k/test-operator-controller-manager-55b5ff4dbb-z5d4k-describe 2026-03-09 19:19:32.022154 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-55b5ff4dbb-z5d4k/logs/ 2026-03-09 19:19:32.022170 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-55b5ff4dbb-z5d4k/logs/manager.log 2026-03-09 19:19:32.022185 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-85b655bd8f-llgvv/ 2026-03-09 19:19:32.022201 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-85b655bd8f-llgvv/watcher-operator-controller-manager-85b655bd8f-llgvv-describe 2026-03-09 19:19:32.022216 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-85b655bd8f-llgvv/logs/ 2026-03-09 19:19:32.022232 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-85b655bd8f-llgvv/logs/manager.log.gz 2026-03-09 19:19:32.022247 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-snmkh/ 2026-03-09 19:19:32.022263 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-snmkh/watcher-operator-index-snmkh-describe 2026-03-09 19:19:32.022278 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-snmkh/logs/ 2026-03-09 19:19:32.022294 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-snmkh/logs/registry-server.log 2026-03-09 19:19:32.022309 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/ 2026-03-09 19:19:32.022325 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/barbican-operator-controller-manager-6db6876945.yaml 2026-03-09 19:19:32.022348 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/cinder-operator-controller-manager-55d77d7b5c.yaml 2026-03-09 19:19:32.022365 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/designate-operator-controller-manager-5d87c9d997.yaml 2026-03-09 19:19:32.022381 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/glance-operator-controller-manager-64db6967f8.yaml 2026-03-09 19:19:32.022397 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/heat-operator-controller-manager-cf99c678f.yaml 2026-03-09 19:19:32.022412 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/horizon-operator-controller-manager-78bc7f9bd9.yaml 2026-03-09 19:19:32.022427 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/infra-operator-controller-manager-f7fcc58b9.yaml 2026-03-09 19:19:32.022443 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/ironic-operator-controller-manager-545456dc4.yaml 2026-03-09 19:19:32.022458 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/keystone-operator-controller-manager-7c789f89c6.yaml 2026-03-09 19:19:32.022473 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/manila-operator-controller-manager-67d996989d.yaml 2026-03-09 19:19:32.022488 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/mariadb-operator-controller-manager-7b6bfb6475.yaml 2026-03-09 19:19:32.022503 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/neutron-operator-controller-manager-54688575f.yaml 2026-03-09 19:19:32.022518 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/nova-operator-controller-manager-74b6b5dc96.yaml 2026-03-09 19:19:32.022534 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/octavia-operator-controller-manager-5d86c7ddb7.yaml 2026-03-09 19:19:32.022553 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-baremetal-operator-controller-manager-7c6767dc9c.yaml 2026-03-09 19:19:32.022569 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-operator-controller-init-787cf98cf6.yaml 2026-03-09 19:19:32.022585 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-operator-controller-manager-64797568c9.yaml 2026-03-09 19:19:32.022600 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/ovn-operator-controller-manager-75684d597f.yaml 2026-03-09 19:19:32.022616 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/placement-operator-controller-manager-648564c9fc.yaml 2026-03-09 19:19:32.022631 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/rabbitmq-cluster-operator-manager-668c99d594.yaml 2026-03-09 19:19:32.022647 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/swift-operator-controller-manager-9b9ff9f4d.yaml 2026-03-09 19:19:32.022662 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/telemetry-operator-controller-manager-5fdb694969.yaml 2026-03-09 19:19:32.022677 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/test-operator-controller-manager-55b5ff4dbb.yaml 2026-03-09 19:19:32.022692 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/watcher-operator-controller-manager-75f555f9d6.yaml 2026-03-09 19:19:32.022707 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/watcher-operator-controller-manager-85b655bd8f.yaml 2026-03-09 19:19:32.022723 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/routes/ 2026-03-09 19:19:32.022738 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ 2026-03-09 19:19:32.022760 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/barbican/ 2026-03-09 19:19:32.022777 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/barbican/barbican-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.022793 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/cinder/ 2026-03-09 19:19:32.022808 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/cinder/cinder-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.022824 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/designate/ 2026-03-09 19:19:32.022840 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/designate/designate-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.022856 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/glance/ 2026-03-09 19:19:32.022872 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/glance/glance-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.022887 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/heat/ 2026-03-09 19:19:32.022903 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/heat/heat-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.022918 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/horizon/ 2026-03-09 19:19:32.022934 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/horizon/horizon-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.022950 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ironic/ 2026-03-09 19:19:32.022965 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ironic/ironic-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.022984 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/keystone/ 2026-03-09 19:19:32.023000 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/keystone/keystone-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023028 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/manila/ 2026-03-09 19:19:32.023049 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/manila/manila-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023066 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/mariadb/ 2026-03-09 19:19:32.023082 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/mariadb/mariadb-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023098 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/neutron/ 2026-03-09 19:19:32.023115 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/neutron/neutron-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023131 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/nova/ 2026-03-09 19:19:32.023147 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/nova/nova-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023162 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/octavia/ 2026-03-09 19:19:32.023178 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/octavia/octavia-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023194 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ovn/ 2026-03-09 19:19:32.023209 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ovn/ovn-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023228 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/placement/ 2026-03-09 19:19:32.023244 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/placement/placement-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023260 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/swift/ 2026-03-09 19:19:32.023275 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/swift/swift-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023291 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/telemetry/ 2026-03-09 19:19:32.023306 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/telemetry/telemetry-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023321 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/ 2026-03-09 19:19:32.023337 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/watcher-operator-controller-manager-service-cert.yaml 2026-03-09 19:19:32.023352 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/watcher-operator-metrics-server-cert.yaml 2026-03-09 19:19:32.023367 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/ 2026-03-09 19:19:32.023383 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/barbican-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023398 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/cinder-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023414 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/designate-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023433 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/glance-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023449 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/heat-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023464 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/horizon-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023479 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/infra-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023495 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/infra-operator-webhook-service.yaml 2026-03-09 19:19:32.023510 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/ironic-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023525 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/keystone-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023555 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/manila-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023571 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/mariadb-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023587 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/neutron-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023605 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/nova-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023620 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/octavia-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023635 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-baremetal-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023653 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-baremetal-operator-webhook-service.yaml 2026-03-09 19:19:32.023668 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023683 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-index.yaml 2026-03-09 19:19:32.023699 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-webhook-service.yaml 2026-03-09 19:19:32.023714 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/ovn-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023729 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/placement-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023745 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/rabbitmq-cluster-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023760 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/swift-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023775 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/telemetry-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023790 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/test-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023805 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-controller-manager-metrics-service.yaml 2026-03-09 19:19:32.023820 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-controller-manager-service.yaml 2026-03-09 19:19:32.023839 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-index.yaml 2026-03-09 19:19:32.023861 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-webhook-service.yaml 2026-03-09 19:19:32.023876 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/statefulset/ 2026-03-09 19:19:32.023892 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/ 2026-03-09 19:19:32.023910 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/openstack-operator 2026-03-09 19:19:32.023926 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/watcher-operator 2026-03-09 19:19:32.023943 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/ 2026-03-09 19:19:32.023959 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/all_resources.log 2026-03-09 19:19:32.023976 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/events.log 2026-03-09 19:19:32.023991 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/nad.log 2026-03-09 19:19:32.024007 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pvc.log 2026-03-09 19:19:32.024039 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/buildconfig/ 2026-03-09 19:19:32.024056 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/cert/ 2026-03-09 19:19:32.024072 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/certificaterequests/ 2026-03-09 19:19:32.024088 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/configmaps/ 2026-03-09 19:19:32.024103 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/cronjobs/ 2026-03-09 19:19:32.024123 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/ 2026-03-09 19:19:32.024138 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/ 2026-03-09 19:19:32.024154 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/ctlplane.yaml 2026-03-09 19:19:32.024169 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/datacentre.yaml 2026-03-09 19:19:32.024186 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/designate.yaml 2026-03-09 19:19:32.024202 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/designateext.yaml 2026-03-09 19:19:32.024218 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/internalapi.yaml 2026-03-09 19:19:32.024234 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/octavia.yaml 2026-03-09 19:19:32.024250 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/storage.yaml 2026-03-09 19:19:32.024266 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/storagemgmt.yaml 2026-03-09 19:19:32.024282 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/tenant.yaml 2026-03-09 19:19:32.024297 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/daemonset/ 2026-03-09 19:19:32.024313 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/deployments/ 2026-03-09 19:19:32.024329 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/imagestream/ 2026-03-09 19:19:32.024348 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/installplans/ 2026-03-09 19:19:32.024364 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/issuer/ 2026-03-09 19:19:32.024379 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/jobs/ 2026-03-09 19:19:32.024395 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/poddisruptionbudgets/ 2026-03-09 19:19:32.024410 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/ 2026-03-09 19:19:32.024425 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/-describe 2026-03-09 19:19:32.024440 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/logs/ 2026-03-09 19:19:32.024455 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/replicaset/ 2026-03-09 19:19:32.024470 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/routes/ 2026-03-09 19:19:32.024485 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/services/ 2026-03-09 19:19:32.024500 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/statefulset/ 2026-03-09 19:19:32.024515 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/subscriptions/ 2026-03-09 19:19:32.024530 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/ 2026-03-09 19:19:32.024545 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/all_resources.log 2026-03-09 19:19:32.024560 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/events.log 2026-03-09 19:19:32.024575 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/nad.log 2026-03-09 19:19:32.024593 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pvc.log 2026-03-09 19:19:32.024609 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/buildconfig/ 2026-03-09 19:19:32.024624 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/ 2026-03-09 19:19:32.024639 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/ceilometer-internal-svc.yaml 2026-03-09 19:19:32.024654 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/cinder-internal-svc.yaml 2026-03-09 19:19:32.024670 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/galera-openstack-svc.yaml 2026-03-09 19:19:32.024685 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-internal-svc.yaml 2026-03-09 19:19:32.024700 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-public-route.yaml 2026-03-09 19:19:32.024715 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-public-svc.yaml 2026-03-09 19:19:32.024730 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/kube-state-metrics-svc.yaml 2026-03-09 19:19:32.024745 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/memcached-mtls.yaml 2026-03-09 19:19:32.024760 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/memcached-svc.yaml 2026-03-09 19:19:32.024775 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/metric-storage-prometheus-svc.yaml 2026-03-09 19:19:32.024790 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rabbitmq-notifications-svc.yaml 2026-03-09 19:19:32.024805 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rabbitmq-svc.yaml 2026-03-09 19:19:32.024824 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-internal.yaml 2026-03-09 19:19:32.024839 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-libvirt.yaml 2026-03-09 19:19:32.024854 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-ovn.yaml 2026-03-09 19:19:32.024869 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-public.yaml 2026-03-09 19:19:32.024884 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/ 2026-03-09 19:19:32.024900 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/ceilometer-internal-svc-1.yaml 2026-03-09 19:19:32.024915 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/cinder-internal-svc-1.yaml 2026-03-09 19:19:32.024931 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/galera-openstack-svc-1.yaml 2026-03-09 19:19:32.024946 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-internal-svc-1.yaml 2026-03-09 19:19:32.024961 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-public-route-1.yaml 2026-03-09 19:19:32.024977 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-public-svc-1.yaml 2026-03-09 19:19:32.024992 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/kube-state-metrics-svc-1.yaml 2026-03-09 19:19:32.025009 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/memcached-mtls-1.yaml 2026-03-09 19:19:32.025047 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/memcached-svc-1.yaml 2026-03-09 19:19:32.025066 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/metric-storage-prometheus-svc-1.yaml 2026-03-09 19:19:32.025085 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rabbitmq-notifications-svc-1.yaml 2026-03-09 19:19:32.025104 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rabbitmq-svc-1.yaml 2026-03-09 19:19:32.025126 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-internal-1.yaml 2026-03-09 19:19:32.025142 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-libvirt-1.yaml 2026-03-09 19:19:32.025158 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-ovn-1.yaml 2026-03-09 19:19:32.025174 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-public-1.yaml 2026-03-09 19:19:32.025189 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/ 2026-03-09 19:19:32.025205 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/appcred-watcher-pre.yaml 2026-03-09 19:19:32.025221 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/infra-instanceha-config.yaml 2026-03-09 19:19:32.025237 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/prometheus-metric-storage-rulefiles-0.yaml 2026-03-09 19:19:32.025252 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-config-data.yaml 2026-03-09 19:19:32.025268 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-config-data.yaml 2026-03-09 19:19:32.025287 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-plugins-conf.yaml 2026-03-09 19:19:32.025304 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-server-conf.yaml 2026-03-09 19:19:32.025319 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-plugins-conf.yaml 2026-03-09 19:19:32.025335 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-server-conf.yaml 2026-03-09 19:19:32.025351 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cronjobs/ 2026-03-09 19:19:32.025367 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cronjobs/keystone-cron.yaml 2026-03-09 19:19:32.025382 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/ 2026-03-09 19:19:32.025398 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/alertmanagers.monitoring.rhobs/ 2026-03-09 19:19:32.025414 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/alertmanagers.monitoring.rhobs/metric-storage.yaml 2026-03-09 19:19:32.025429 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/ceilometers.telemetry.openstack.org/ 2026-03-09 19:19:32.025445 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/ceilometers.telemetry.openstack.org/ceilometer.yaml 2026-03-09 19:19:32.025461 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/galeras.mariadb.openstack.org/ 2026-03-09 19:19:32.025476 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/galeras.mariadb.openstack.org/openstack.yaml 2026-03-09 19:19:32.025492 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneapis.keystone.openstack.org/ 2026-03-09 19:19:32.025522 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneapis.keystone.openstack.org/keystone.yaml 2026-03-09 19:19:32.025538 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneservices.keystone.openstack.org/ 2026-03-09 19:19:32.025553 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneservices.keystone.openstack.org/ceilometer.yaml 2026-03-09 19:19:32.025568 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/ 2026-03-09 19:19:32.025584 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/keystone.yaml 2026-03-09 19:19:32.025599 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/openstack-mariadb-root.yaml 2026-03-09 19:19:32.025615 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbdatabases.mariadb.openstack.org/ 2026-03-09 19:19:32.025631 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbdatabases.mariadb.openstack.org/keystone.yaml 2026-03-09 19:19:32.025647 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/memcacheds.memcached.openstack.org/ 2026-03-09 19:19:32.025662 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/memcacheds.memcached.openstack.org/memcached.yaml 2026-03-09 19:19:32.025678 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/metricstorages.telemetry.openstack.org/ 2026-03-09 19:19:32.025693 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/metricstorages.telemetry.openstack.org/metric-storage.yaml 2026-03-09 19:19:32.025709 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/monitoringstacks.monitoring.rhobs/ 2026-03-09 19:19:32.025728 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/monitoringstacks.monitoring.rhobs/metric-storage.yaml 2026-03-09 19:19:32.025744 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackclients.client.openstack.org/ 2026-03-09 19:19:32.025759 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackclients.client.openstack.org/openstackclient.yaml 2026-03-09 19:19:32.025775 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackcontrolplanes.core.openstack.org/ 2026-03-09 19:19:32.025790 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackcontrolplanes.core.openstack.org/openstack.yaml 2026-03-09 19:19:32.025806 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackversions.core.openstack.org/ 2026-03-09 19:19:32.025821 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackversions.core.openstack.org/openstack.yaml 2026-03-09 19:19:32.025837 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheuses.monitoring.rhobs/ 2026-03-09 19:19:32.025852 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheuses.monitoring.rhobs/metric-storage.yaml 2026-03-09 19:19:32.025867 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheusrules.monitoring.rhobs/ 2026-03-09 19:19:32.025883 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheusrules.monitoring.rhobs/metric-storage.yaml 2026-03-09 19:19:32.025898 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/ 2026-03-09 19:19:32.025912 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/rabbitmq-notifications.yaml 2026-03-09 19:19:32.025931 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/rabbitmq.yaml 2026-03-09 19:19:32.025946 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/ 2026-03-09 19:19:32.025961 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/rabbitmq-notifications.yaml 2026-03-09 19:19:32.025976 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/rabbitmq.yaml 2026-03-09 19:19:32.025991 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/ 2026-03-09 19:19:32.026007 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-compute-prom-exporter-tls.yaml 2026-03-09 19:19:32.026041 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-compute-prom-exporter.yaml 2026-03-09 19:19:32.026058 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-ipmi-prom-exporter-tls.yaml 2026-03-09 19:19:32.026074 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-ipmi-prom-exporter.yaml 2026-03-09 19:19:32.026089 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer.yaml 2026-03-09 19:19:32.026104 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-kepler.yaml 2026-03-09 19:19:32.026120 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ksm.yaml 2026-03-09 19:19:32.026135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-node-exporter-tls.yaml 2026-03-09 19:19:32.026157 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-node-exporter.yaml 2026-03-09 19:19:32.026174 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-openstack-network-exporter-tls.yaml 2026-03-09 19:19:32.026189 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-openstack-network-exporter.yaml 2026-03-09 19:19:32.026205 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-podman-exporter-tls.yaml 2026-03-09 19:19:32.026221 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-podman-exporter.yaml 2026-03-09 19:19:32.026236 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-rabbitmq-notifications.yaml 2026-03-09 19:19:32.026252 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-rabbitmq.yaml 2026-03-09 19:19:32.026267 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/telemetries.telemetry.openstack.org/ 2026-03-09 19:19:32.026282 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/telemetries.telemetry.openstack.org/telemetry.yaml 2026-03-09 19:19:32.026298 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/ 2026-03-09 19:19:32.026313 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/ceilometer-notifications-transport.yaml 2026-03-09 19:19:32.026329 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/keystone-keystone-transport.yaml 2026-03-09 19:19:32.026344 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/daemonset/ 2026-03-09 19:19:32.026363 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/deployments/ 2026-03-09 19:19:32.026379 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/deployments/keystone.yaml 2026-03-09 19:19:32.026396 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/imagestream/ 2026-03-09 19:19:32.026412 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/installplans/ 2026-03-09 19:19:32.026428 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/ 2026-03-09 19:19:32.026443 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-internal.yaml 2026-03-09 19:19:32.026459 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-libvirt.yaml 2026-03-09 19:19:32.026474 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-ovn.yaml 2026-03-09 19:19:32.026490 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-public.yaml 2026-03-09 19:19:32.026505 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/selfsigned-issuer.yaml 2026-03-09 19:19:32.026521 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/jobs/ 2026-03-09 19:19:32.026536 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/jobs/keystone-cron-29551381.yaml 2026-03-09 19:19:32.026552 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/poddisruptionbudgets/ 2026-03-09 19:19:32.026568 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/poddisruptionbudgets/metric-storage-alertmanager.yaml 2026-03-09 19:19:32.026584 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ 2026-03-09 19:19:32.026603 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/ 2026-03-09 19:19:32.026619 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/alertmanager-metric-storage-0-describe 2026-03-09 19:19:32.026635 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/ 2026-03-09 19:19:32.026651 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/alertmanager.log 2026-03-09 19:19:32.026667 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/config-reloader.log 2026-03-09 19:19:32.026682 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/init-config-reloader.log 2026-03-09 19:19:32.026698 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/ 2026-03-09 19:19:32.026713 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/ceilometer-0-describe 2026-03-09 19:19:32.026728 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ 2026-03-09 19:19:32.026743 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ceilometer-central-agent.log 2026-03-09 19:19:32.026758 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ceilometer-notification-agent.log 2026-03-09 19:19:32.026772 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/proxy-httpd.log 2026-03-09 19:19:32.026788 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/sg-core.log 2026-03-09 19:19:32.026806 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-6d45c85556-w6k7b/ 2026-03-09 19:19:32.026822 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-6d45c85556-w6k7b/keystone-6d45c85556-w6k7b-describe 2026-03-09 19:19:32.026838 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-6d45c85556-w6k7b/logs/ 2026-03-09 19:19:32.026853 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-6d45c85556-w6k7b/logs/keystone-api.log 2026-03-09 19:19:32.026868 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-cron-29551381-c5bp4/ 2026-03-09 19:19:32.026883 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-cron-29551381-c5bp4/keystone-cron-29551381-c5bp4-describe 2026-03-09 19:19:32.026898 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-cron-29551381-c5bp4/logs/ 2026-03-09 19:19:32.026914 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-cron-29551381-c5bp4/logs/keystone-cron.log 2026-03-09 19:19:32.026929 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/ 2026-03-09 19:19:32.026945 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/kube-state-metrics-0-describe 2026-03-09 19:19:32.026960 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/logs/ 2026-03-09 19:19:32.026975 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/logs/kube-state-metrics.log 2026-03-09 19:19:32.026991 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/ 2026-03-09 19:19:32.027006 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/memcached-0-describe 2026-03-09 19:19:32.027043 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/logs/ 2026-03-09 19:19:32.027061 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/logs/memcached.log.gz 2026-03-09 19:19:32.027077 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/ 2026-03-09 19:19:32.027092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/openstack-galera-0-describe 2026-03-09 19:19:32.027108 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/ 2026-03-09 19:19:32.027123 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/galera.log 2026-03-09 19:19:32.027139 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/mysql-bootstrap.log 2026-03-09 19:19:32.027154 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/ 2026-03-09 19:19:32.027170 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/openstackclient-describe 2026-03-09 19:19:32.027185 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/logs/ 2026-03-09 19:19:32.027201 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/logs/openstackclient.log 2026-03-09 19:19:32.027216 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/ 2026-03-09 19:19:32.027232 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/prometheus-metric-storage-0-describe 2026-03-09 19:19:32.027251 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/ 2026-03-09 19:19:32.027268 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/config-reloader.log 2026-03-09 19:19:32.027284 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/init-config-reloader.log 2026-03-09 19:19:32.027300 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/prometheus.log 2026-03-09 19:19:32.027316 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/thanos-sidecar.log 2026-03-09 19:19:32.027332 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/ 2026-03-09 19:19:32.027347 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/rabbitmq-notifications-server-0-describe 2026-03-09 19:19:32.027363 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/ 2026-03-09 19:19:32.027378 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/rabbitmq.log 2026-03-09 19:19:32.027393 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/setup-container.log 2026-03-09 19:19:32.027408 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/ 2026-03-09 19:19:32.027423 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/rabbitmq-server-0-describe 2026-03-09 19:19:32.027438 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/ 2026-03-09 19:19:32.027453 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/rabbitmq.log 2026-03-09 19:19:32.027472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/setup-container.log 2026-03-09 19:19:32.027487 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/ 2026-03-09 19:19:32.027503 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/keystone-6d45c85556.yaml 2026-03-09 19:19:32.027519 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/keystone-7774c4794c.yaml 2026-03-09 19:19:32.027553 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/routes/ 2026-03-09 19:19:32.027571 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/routes/keystone-public.yaml 2026-03-09 19:19:32.027587 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ 2026-03-09 19:19:32.027606 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ 2026-03-09 19:19:32.027625 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml 2026-03-09 19:19:32.027641 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml-ceilometer.conf 2026-03-09 19:19:32.027657 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml-custom.conf 2026-03-09 19:19:32.027672 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-scripts.yaml 2026-03-09 19:19:32.027687 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml 2026-03-09 19:19:32.027703 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-ceilometer.conf 2026-03-09 19:19:32.027723 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-custom.conf 2026-03-09 19:19:32.027739 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-httpd.conf 2026-03-09 19:19:32.027755 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-ssl.conf 2026-03-09 19:19:32.027771 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml 2026-03-09 19:19:32.027786 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml-ceilometer.conf 2026-03-09 19:19:32.027802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml-custom.conf 2026-03-09 19:19:32.027818 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-scripts.yaml 2026-03-09 19:19:32.027833 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-scripts.yaml 2026-03-09 19:19:32.027849 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/cert-ceilometer-internal-svc.yaml 2026-03-09 19:19:32.027864 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/rabbitmq-transport-url-ceilometer-notifications-transport.yaml 2026-03-09 19:19:32.027880 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/ 2026-03-09 19:19:32.027896 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-internal-svc.yaml 2026-03-09 19:19:32.027911 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-public-route.yaml 2026-03-09 19:19:32.027930 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-public-svc.yaml 2026-03-09 19:19:32.027946 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/ 2026-03-09 19:19:32.027961 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-internal-svc.yaml 2026-03-09 19:19:32.027976 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-public-route.yaml 2026-03-09 19:19:32.027991 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-public-svc.yaml 2026-03-09 19:19:32.028007 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml 2026-03-09 19:19:32.028040 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-httpd.conf 2026-03-09 19:19:32.028058 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-keystone.conf 2026-03-09 19:19:32.028076 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-ssl.conf 2026-03-09 19:19:32.028094 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-db-secret.yaml 2026-03-09 19:19:32.028110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-scripts.yaml 2026-03-09 19:19:32.028125 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone.yaml 2026-03-09 19:19:32.028141 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/rabbitmq-transport-url-keystone-keystone-transport.yaml 2026-03-09 19:19:32.028160 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/mariadb/ 2026-03-09 19:19:32.028176 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/mariadb/openstack-mariadb-root-db-secret.yaml 2026-03-09 19:19:32.028192 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/ 2026-03-09 19:19:32.028208 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-cluster-tls-config.yaml 2026-03-09 19:19:32.028224 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-generated.yaml 2026-03-09 19:19:32.028240 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-tls-assets-0.yaml 2026-03-09 19:19:32.028256 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-web-config.yaml 2026-03-09 19:19:32.028272 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/cert-metric-storage-prometheus-svc.yaml 2026-03-09 19:19:32.028287 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/metric-storage-prometheus-endpoint.yaml 2026-03-09 19:19:32.028303 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/metric-storage-self-scrape.yaml 2026-03-09 19:19:32.028318 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-thanos-prometheus-http-client-file.yaml 2026-03-09 19:19:32.028334 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-tls-assets-0.yaml 2026-03-09 19:19:32.028349 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-web-config.yaml 2026-03-09 19:19:32.028368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage.yaml 2026-03-09 19:19:32.028385 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ovn/ 2026-03-09 19:19:32.028400 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ovn/rootca-ovn.yaml 2026-03-09 19:19:32.028416 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/ 2026-03-09 19:19:32.028432 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/cert-rabbitmq-notifications-svc.yaml 2026-03-09 19:19:32.028448 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/cert-rabbitmq-svc.yaml 2026-03-09 19:19:32.028463 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-default-user.yaml 2026-03-09 19:19:32.028479 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-default-user.yaml-default_user.conf 2026-03-09 19:19:32.028494 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-erlang-cookie.yaml 2026-03-09 19:19:32.028510 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-default-user.yaml 2026-03-09 19:19:32.028525 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-default-user.yaml-default_user.conf 2026-03-09 19:19:32.028541 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-erlang-cookie.yaml 2026-03-09 19:19:32.028556 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-transport-url-ceilometer-notifications-transport.yaml 2026-03-09 19:19:32.028572 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-transport-url-keystone-keystone-transport.yaml 2026-03-09 19:19:32.028590 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/ 2026-03-09 19:19:32.028606 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/ac-watcher-secret.yaml 2026-03-09 19:19:32.028622 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/cert-watcher-internal-svc.yaml 2026-03-09 19:19:32.028638 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/cert-watcher-public-svc.yaml 2026-03-09 19:19:32.028654 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/ 2026-03-09 19:19:32.028670 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/alertmanager-operated.yaml 2026-03-09 19:19:32.028686 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/ceilometer-internal.yaml 2026-03-09 19:19:32.028702 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/keystone-internal.yaml 2026-03-09 19:19:32.028717 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/keystone-public.yaml 2026-03-09 19:19:32.028732 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/kube-state-metrics.yaml 2026-03-09 19:19:32.028748 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/memcached.yaml 2026-03-09 19:19:32.028763 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-alertmanager.yaml 2026-03-09 19:19:32.028778 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-prometheus.yaml 2026-03-09 19:19:32.028798 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-thanos-sidecar.yaml 2026-03-09 19:19:32.028814 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/openstack-galera.yaml 2026-03-09 19:19:32.028829 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/openstack.yaml 2026-03-09 19:19:32.028845 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/prometheus-operated.yaml 2026-03-09 19:19:32.028860 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-nodes.yaml 2026-03-09 19:19:32.028875 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-notifications-nodes.yaml 2026-03-09 19:19:32.028891 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-notifications.yaml 2026-03-09 19:19:32.028906 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq.yaml 2026-03-09 19:19:32.028921 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/ 2026-03-09 19:19:32.028936 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/alertmanager-metric-storage.yaml 2026-03-09 19:19:32.028952 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/ceilometer.yaml 2026-03-09 19:19:32.028967 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/kube-state-metrics.yaml 2026-03-09 19:19:32.028982 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/memcached.yaml 2026-03-09 19:19:32.028997 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/openstack-galera.yaml 2026-03-09 19:19:32.029026 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/prometheus-metric-storage.yaml 2026-03-09 19:19:32.029050 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/rabbitmq-notifications-server.yaml 2026-03-09 19:19:32.029067 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/rabbitmq-server.yaml 2026-03-09 19:19:32.029082 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/subscriptions/ 2026-03-09 19:19:32.029098 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ 2026-03-09 19:19:32.029114 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/l2advertisement.yaml 2026-03-09 19:19:32.029130 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/ 2026-03-09 19:19:32.029145 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/ctlplane.yaml 2026-03-09 19:19:32.029161 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/designateext.yaml 2026-03-09 19:19:32.029177 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/internalapi.yaml 2026-03-09 19:19:32.029193 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/storage.yaml 2026-03-09 19:19:32.029208 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/tenant.yaml 2026-03-09 19:19:32.029223 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nnce/ 2026-03-09 19:19:32.029239 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nnce/crc.ens7-crc.yaml 2026-03-09 19:19:32.029254 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nncp/ 2026-03-09 19:19:32.029270 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nncp/ens7-crc.yaml 2026-03-09 19:19:32.029289 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nns/ 2026-03-09 19:19:32.029305 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nns/crc.yaml 2026-03-09 19:19:32.029320 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/ 2026-03-09 19:19:32.029336 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/crc.describe 2026-03-09 19:19:32.029351 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/crc.yaml 2026-03-09 19:19:32.029366 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/ 2026-03-09 19:19:32.029382 | controller | cL+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/sos-report- -> ../_all_nodes/sosreport- 2026-03-09 19:19:32.029397 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/_all_nodes/ 2026-03-09 19:19:32.029412 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/ 2026-03-09 19:19:32.029428 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/ 2026-03-09 19:19:32.029444 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/cert-manager-webhook.yaml 2026-03-09 19:19:32.029459 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/infra-operator-mutating-webhook-configuration.yaml 2026-03-09 19:19:32.029475 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcher-v1beta1.kb.io-g4b5h.yaml 2026-03-09 19:19:32.029491 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherapi-v1beta1.kb.io-7lbbn.yaml 2026-03-09 19:19:32.029507 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherapplier-v1beta1.kb.io-87q82.yaml 2026-03-09 19:19:32.029524 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherdecisionengine-v1beta1.kb.io-xhtmm.yaml 2026-03-09 19:19:32.029546 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/nmstate.yaml 2026-03-09 19:19:32.029563 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/openstack-baremetal-operator-mutating-webhook-configuration.yaml 2026-03-09 19:19:32.029578 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/openstack-operator-mutating-webhook-configuration.yaml 2026-03-09 19:19:32.029594 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/ 2026-03-09 19:19:32.029610 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/alertmanagerconfigs.monitoring.rhobs-s56dz.yaml 2026-03-09 19:19:32.029625 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bfdprofilevalidationwebhook.metallb.io-lp6dc.yaml 2026-03-09 19:19:32.029641 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bgpadvertisementvalidationwebhook.metallb.io-zsm5s.yaml 2026-03-09 19:19:32.029656 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bgppeersvalidationwebhook.metallb.io-5dvng.yaml 2026-03-09 19:19:32.029671 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/cert-manager-webhook.yaml 2026-03-09 19:19:32.029686 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/communityvalidationwebhook.metallb.io-wlj8p.yaml 2026-03-09 19:19:32.029702 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/controlplanemachineset.machine.openshift.io.yaml 2026-03-09 19:19:32.029717 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/frr-k8s-validating-webhook-configuration.yaml 2026-03-09 19:19:32.029733 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/infra-operator-validating-webhook-configuration.yaml 2026-03-09 19:19:32.029749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/ipaddresspoolvalidationwebhook.metallb.io-n62h4.yaml 2026-03-09 19:19:32.029769 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/l2advertisementvalidationwebhook.metallb.io-lcxkm.yaml 2026-03-09 19:19:32.029785 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/metallbvalidationwebhook.metallb.io-ks47k.yaml 2026-03-09 19:19:32.029802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/monitoringconfigmaps.openshift.io.yaml 2026-03-09 19:19:32.029817 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/multus.openshift.io.yaml 2026-03-09 19:19:32.029833 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/network-node-identity.openshift.io.yaml 2026-03-09 19:19:32.029848 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/openstack-baremetal-operator-validating-webhook-configuration.yaml 2026-03-09 19:19:32.029864 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/openstack-operator-validating-webhook-configuration.yaml 2026-03-09 19:19:32.029879 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/prometheusrules.monitoring.rhobs-5dpq6.yaml 2026-03-09 19:19:32.029958 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcher-v1beta1.kb.io-2f6kq.yaml 2026-03-09 19:19:32.029983 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherapi-v1beta1.kb.io-qtrh7.yaml 2026-03-09 19:19:32.030001 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherapplier-v1beta1.kb.io-vh69p.yaml 2026-03-09 19:19:32.030035 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherdecisionengine-v1beta1.kb.io-tprwq.yaml 2026-03-09 19:19:32.030054 | controller | cd+++++++++ registries.conf.d/ 2026-03-09 19:19:32.030072 | controller | >f+++++++++ registries.conf.d/000-shortnames.conf 2026-03-09 19:19:32.030089 | controller | cd+++++++++ system-config/ 2026-03-09 19:19:32.030106 | controller | cd+++++++++ system-config/libvirt/ 2026-03-09 19:19:32.030123 | controller | >f+++++++++ system-config/libvirt/libvirt-admin.conf 2026-03-09 19:19:32.030145 | controller | >f+++++++++ system-config/libvirt/libvirt.conf 2026-03-09 19:19:32.503609 | controller | ok 2026-03-09 19:19:33.068047 | controller | ok 2026-03-09 19:19:33.176561 | 2026-03-09 19:19:33.176693 | TASK [Return artifact to Zuul] 2026-03-09 19:19:33.203570 | controller | skipping: Conditional result was False 2026-03-09 19:19:33.377136 | 2026-03-09 19:19:33.377407 | PLAY [Run ci/playbooks/collect-logs.yml on CRC host] 2026-03-09 19:19:33.424289 | 2026-03-09 19:19:33.424420 | TASK [Get kubelet journalctl logs] 2026-03-09 19:19:34.473558 | crc | Output suppressed because no_log was given 2026-03-09 19:19:34.488548 | 2026-03-09 19:19:34.488685 | TASK [Compress logs bigger than 2MB] 2026-03-09 19:19:34.770419 | crc | gzip: /var/home/core/zuul-output/logs/kubelet.log.gz already exists; not overwritten 2026-03-09 19:19:35.051141 | crc | ERROR 2026-03-09 19:19:35.051378 | crc | { 2026-03-09 19:19:35.051439 | crc | "delta": "0:00:00.007349", 2026-03-09 19:19:35.051481 | crc | "end": "2026-03-09 19:19:34.770964", 2026-03-09 19:19:35.051522 | crc | "msg": "non-zero return code", 2026-03-09 19:19:35.051559 | crc | "rc": 1, 2026-03-09 19:19:35.051596 | crc | "start": "2026-03-09 19:19:34.763615" 2026-03-09 19:19:35.051632 | crc | } 2026-03-09 19:19:35.051682 | crc | ERROR: Ignoring Errors 2026-03-09 19:19:35.067865 | 2026-03-09 19:19:35.067994 | TASK [Copy files from workspace on node] 2026-03-09 19:19:35.103308 | crc | ok 2026-03-09 19:19:35.135902 | 2026-03-09 19:19:35.136003 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-09 19:19:35.173401 | crc | ok 2026-03-09 19:19:35.182875 | 2026-03-09 19:19:35.182963 | TASK [fetch-output : Set log path for single node] 2026-03-09 19:19:35.208078 | crc | skipping: Conditional result was False 2026-03-09 19:19:35.218509 | 2026-03-09 19:19:35.218606 | LOOP [fetch-output : Ensure local output dirs] 2026-03-09 19:19:35.487772 | crc -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/logs/crc" 2026-03-09 19:19:35.488073 | crc -> localhost | changed: All items complete 2026-03-09 19:19:35.488122 | 2026-03-09 19:19:35.732058 | crc -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/artifacts" 2026-03-09 19:19:35.926725 | crc -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/docs" 2026-03-09 19:19:35.945380 | 2026-03-09 19:19:35.945510 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-09 19:19:36.666352 | crc | changed: 2026-03-09 19:19:36.667002 | crc | .d..t...... ./ 2026-03-09 19:19:36.667278 | crc | >f+++++++++ kubelet.log 2026-03-09 19:19:37.232534 | crc | ok 2026-03-09 19:19:37.788667 | crc | ok 2026-03-09 19:19:37.854832 | 2026-03-09 19:19:37.854956 | PLAY RECAP 2026-03-09 19:19:37.855041 | controller | ok: 15 changed: 11 unreachable: 0 failed: 0 skipped: 7 rescued: 0 ignored: 1 2026-03-09 19:19:37.855086 | crc | ok: 5 changed: 3 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 1 2026-03-09 19:19:37.855113 | 2026-03-09 19:19:37.998781 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml@main] 2026-03-09 19:19:38.008236 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml@main] 2026-03-09 19:19:38.612784 | 2026-03-09 19:19:38.612934 | PLAY [Run ci/playbooks/multinode-autohold.yml] 2026-03-09 19:19:38.691371 | 2026-03-09 19:19:38.691511 | TASK [Verify if "success" flag exists after successful tests execution] 2026-03-09 19:19:39.236133 | controller | ok 2026-03-09 19:19:39.260122 | 2026-03-09 19:19:39.260225 | TASK [Fetch existing autoholds from zuul] 2026-03-09 19:19:39.339076 | controller | ok 2026-03-09 19:19:39.390661 | 2026-03-09 19:19:39.390760 | TASK [krb_request : Perform the request] 2026-03-09 19:19:40.028270 | controller | ok 2026-03-09 19:19:40.115432 | 2026-03-09 19:19:40.115557 | TASK [Check if any autohold matches] 2026-03-09 19:19:40.191499 | controller | skipping: Conditional result was False 2026-03-09 19:19:40.316779 | 2026-03-09 19:19:40.316891 | PLAY RECAP 2026-03-09 19:19:40.316938 | controller | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0 2026-03-09 19:19:40.316965 | 2026-03-09 19:19:40.428942 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml@main] 2026-03-09 19:19:40.439377 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml@master] 2026-03-09 19:19:41.051955 | 2026-03-09 19:19:41.052082 | PLAY [crc] 2026-03-09 19:19:41.079045 | 2026-03-09 19:19:41.079131 | TASK [Print machineconfigpool] 2026-03-09 19:19:41.908545 | crc | NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE 2026-03-09 19:19:41.908623 | crc | master rendered-master-f83a149c7490c3b0bf850df839837383 False False True 1 0 0 1 379d 2026-03-09 19:19:41.908635 | crc | worker rendered-worker-2288145f2e2a63a0af5ce658fbeaf678 True False False 0 0 0 0 379d 2026-03-09 19:19:42.117298 | crc | changed 2026-03-09 19:19:42.172486 | 2026-03-09 19:19:42.172566 | PLAY RECAP 2026-03-09 19:19:42.172622 | crc | ok: 1 changed: 1 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-09 19:19:42.172651 | 2026-03-09 19:19:42.274079 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml@master] 2026-03-09 19:19:42.283784 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/compress-crc-cloud-logs.yaml@master] 2026-03-09 19:19:42.863137 | 2026-03-09 19:19:42.863242 | PLAY [crc] 2026-03-09 19:19:42.889903 | 2026-03-09 19:19:42.890030 | TASK [Create compressed log tar gz file - crc-cloud logs] 2026-03-09 19:19:43.606189 | crc | tar: Removing leading `/' from member names 2026-03-09 19:19:43.606243 | crc | tar: /var/home/core/workdir/*log: Cannot stat: No such file or directory 2026-03-09 19:19:43.608366 | crc | tar: Exiting with failure status due to previous errors 2026-03-09 19:19:43.928737 | crc | ok 2026-03-09 19:19:43.945153 | 2026-03-09 19:19:43.945295 | TASK [Copy crc-cloud logs] 2026-03-09 19:19:44.428407 | crc | cp: cannot stat '/var/home/core/workdir/*log': No such file or directory 2026-03-09 19:19:44.510234 | crc | ok 2026-03-09 19:19:44.560405 | 2026-03-09 19:19:44.560513 | PLAY RECAP 2026-03-09 19:19:44.560572 | crc | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-09 19:19:44.560599 | 2026-03-09 19:19:44.661397 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/compress-crc-cloud-logs.yaml@master] 2026-03-09 19:19:44.672320 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/compress-logs.yaml@master] 2026-03-09 19:19:45.291383 | 2026-03-09 19:19:45.291487 | PLAY [all] 2026-03-09 19:19:45.403477 | 2026-03-09 19:19:45.403622 | TASK [Create compressed log tar gz file - ci-framework-data] 2026-03-09 19:19:46.085204 | controller | tar: Removing leading `/' from member names 2026-03-09 19:19:46.209482 | crc | tar: Removing leading `/' from member names 2026-03-09 19:19:46.212496 | crc | tar: /var/home/core/ci-framework-data: Cannot stat: No such file or directory 2026-03-09 19:19:46.212553 | crc | tar: Exiting with failure status due to previous errors 2026-03-09 19:19:46.214585 | crc | mv: cannot move '/var/home/core/ci-framework-data-crc-all-logs.tar.gz' to '/var/home/core/ci-framework-data/logs/ci-framework-data-crc-all-logs.tar.gz': No such file or directory 2026-03-09 19:19:46.216192 | crc | chown: cannot access '/var/home/core/ci-framework-data/logs/ci-framework-data-crc-all-logs.tar.gz': No such file or directory 2026-03-09 19:19:46.557277 | crc | ERROR 2026-03-09 19:19:46.557655 | crc | { 2026-03-09 19:19:46.557723 | crc | "delta": "0:00:00.014241", 2026-03-09 19:19:46.557769 | crc | "end": "2026-03-09 19:19:46.216716", 2026-03-09 19:19:46.557808 | crc | "msg": "non-zero return code", 2026-03-09 19:19:46.557855 | crc | "rc": 1, 2026-03-09 19:19:46.557894 | crc | "start": "2026-03-09 19:19:46.202475" 2026-03-09 19:19:46.557933 | crc | } 2026-03-09 19:19:46.558000 | crc | ERROR: Ignoring Errors 2026-03-09 19:19:54.871453 | controller | ok 2026-03-09 19:19:54.902844 | 2026-03-09 19:19:54.902946 | TASK [Create compressed log tar gz file - zuul-output] 2026-03-09 19:19:55.361049 | controller | tar: Removing leading `/' from member names 2026-03-09 19:19:55.402367 | crc | tar: Removing leading `/' from member names 2026-03-09 19:19:56.081273 | crc | ok 2026-03-09 19:19:58.046462 | controller | ok 2026-03-09 19:19:58.222876 | 2026-03-09 19:19:58.222982 | PLAY RECAP 2026-03-09 19:19:58.223053 | controller | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-09 19:19:58.223083 | crc | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 1 2026-03-09 19:19:58.223102 | 2026-03-09 19:19:58.329622 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/compress-logs.yaml@master] 2026-03-09 19:19:58.339838 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup.yml@master] 2026-03-09 19:19:58.970999 | 2026-03-09 19:19:58.971154 | PLAY [Run playbooks/crc/bootstraped-ci-network-cleanup.yml] 2026-03-09 19:19:59.056477 | 2026-03-09 19:19:59.056613 | TASK [Create openstack config dir] 2026-03-09 19:19:59.618960 | controller | ok 2026-03-09 19:19:59.643985 | 2026-03-09 19:19:59.644093 | TASK [Check for cleanup skip flag file presence] 2026-03-09 19:20:00.010094 | controller | ok 2026-03-09 19:20:00.043312 | 2026-03-09 19:20:00.043390 | TASK [Generate clouds config from cloud_secrets secret] 2026-03-09 19:20:01.015297 | controller | changed 2026-03-09 19:20:01.047571 | 2026-03-09 19:20:01.047685 | LOOP [Delete ports from each host] 2026-03-09 19:20:01.223621 | controller | ok: "crc" 2026-03-09 19:20:04.163199 | controller | ok: "controller" 2026-03-09 19:20:04.233502 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup-instance-ports.yml 2026-03-09 19:20:04.238359 | controller | included: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup-instance-ports.yml 2026-03-09 19:20:04.276972 | 2026-03-09 19:20:04.277137 | TASK [Fetch port info] 2026-03-09 19:20:06.712599 | controller | admin_state_up: true 2026-03-09 19:20:06.712706 | controller | allowed_address_pairs: [] 2026-03-09 19:20:06.712872 | controller | binding_host_id: null 2026-03-09 19:20:06.712977 | controller | binding_profile: {} 2026-03-09 19:20:06.713075 | controller | binding_vif_details: {} 2026-03-09 19:20:06.713189 | controller | binding_vif_type: null 2026-03-09 19:20:06.713315 | controller | binding_vnic_type: normal 2026-03-09 19:20:06.713428 | controller | created_at: '2026-03-09T18:22:33Z' 2026-03-09 19:20:06.713526 | controller | data_plane_status: null 2026-03-09 19:20:06.713616 | controller | description: '' 2026-03-09 19:20:06.713724 | controller | device_id: ea3e2df3-251a-4cdb-9064-3c52ac509aba 2026-03-09 19:20:06.713819 | controller | device_owner: compute:nova 2026-03-09 19:20:06.713912 | controller | device_profile: null 2026-03-09 19:20:06.714210 | controller | dns_assignment: 2026-03-09 19:20:06.714302 | controller | - fqdn: host-192-168-122-10.openstacklocal. 2026-03-09 19:20:06.714309 | controller | hostname: host-192-168-122-10 2026-03-09 19:20:06.714315 | controller | ip_address: 192.168.122.10 2026-03-09 19:20:06.714322 | controller | dns_domain: '' 2026-03-09 19:20:06.714391 | controller | dns_name: '' 2026-03-09 19:20:06.714482 | controller | extra_dhcp_opts: [] 2026-03-09 19:20:06.714709 | controller | fixed_ips: 2026-03-09 19:20:06.714832 | controller | - ip_address: 192.168.122.10 2026-03-09 19:20:06.714841 | controller | subnet_id: edcfb140-8934-4cc7-a46a-d8cf20ecfdf7 2026-03-09 19:20:06.714847 | controller | hardware_offload_type: null 2026-03-09 19:20:06.714918 | controller | hints: '' 2026-03-09 19:20:06.715046 | controller | id: 2a0fed40-01ed-4d11-863e-67f7d40991d7 2026-03-09 19:20:06.715157 | controller | ip_allocation: immediate 2026-03-09 19:20:06.715267 | controller | mac_address: fa:16:3e:d3:67:c2 2026-03-09 19:20:06.715384 | controller | name: crc-ea3e2df3-251a-4cdb-9064-3c52ac509aba 2026-03-09 19:20:06.715519 | controller | network_id: d09b02e2-af90-41f7-a8a9-5064393c43d0 2026-03-09 19:20:06.715625 | controller | numa_affinity_policy: null 2026-03-09 19:20:06.715751 | controller | port_security_enabled: false 2026-03-09 19:20:06.715867 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-09 19:20:06.715964 | controller | propagate_uplink_status: null 2026-03-09 19:20:06.716059 | controller | resource_request: null 2026-03-09 19:20:06.716221 | controller | revision_number: 4 2026-03-09 19:20:06.716325 | controller | qos_network_policy_id: null 2026-03-09 19:20:06.716420 | controller | qos_policy_id: null 2026-03-09 19:20:06.716518 | controller | security_group_ids: [] 2026-03-09 19:20:06.716628 | controller | status: ACTIVE 2026-03-09 19:20:06.716743 | controller | tags: [] 2026-03-09 19:20:06.716839 | controller | trunk_details: null 2026-03-09 19:20:06.716928 | controller | trusted: null 2026-03-09 19:20:06.717035 | controller | updated_at: '2026-03-09T18:22:48Z' 2026-03-09 19:20:06.935188 | controller | changed 2026-03-09 19:20:06.969300 | 2026-03-09 19:20:06.969434 | TASK [Set port info yaml for further usage] 2026-03-09 19:20:07.069866 | controller | ok 2026-03-09 19:20:07.128667 | 2026-03-09 19:20:07.128796 | TASK [Detach port] 2026-03-09 19:20:09.752386 | controller | changed 2026-03-09 19:20:09.826948 | 2026-03-09 19:20:09.827090 | LOOP [Delete port association with trunk network] 2026-03-09 19:20:09.928005 | 2026-03-09 19:20:09.928200 | LOOP [Delete trunk sub ports (tagged ones)] 2026-03-09 19:20:10.011196 | 2026-03-09 19:20:10.011420 | TASK [Delete the trunk network itself] 2026-03-09 19:20:10.674798 | controller | skipping: Conditional result was False 2026-03-09 19:20:10.730165 | 2026-03-09 19:20:10.730332 | TASK [Remove the parent/trunk port itself] 2026-03-09 19:20:24.495178 | controller | changed 2026-03-09 19:20:24.527483 | 2026-03-09 19:20:24.527578 | TASK [Fetch port info] 2026-03-09 19:20:36.836629 | controller | admin_state_up: true 2026-03-09 19:20:36.836746 | controller | allowed_address_pairs: [] 2026-03-09 19:20:36.836921 | controller | binding_host_id: null 2026-03-09 19:20:36.837013 | controller | binding_profile: {} 2026-03-09 19:20:36.837140 | controller | binding_vif_details: {} 2026-03-09 19:20:36.837271 | controller | binding_vif_type: null 2026-03-09 19:20:36.837417 | controller | binding_vnic_type: normal 2026-03-09 19:20:36.837552 | controller | created_at: '2026-03-09T18:22:19Z' 2026-03-09 19:20:36.837644 | controller | data_plane_status: null 2026-03-09 19:20:36.837747 | controller | description: '' 2026-03-09 19:20:36.837887 | controller | device_id: 593e5d4e-760b-40a9-aa62-62e7bf2adfde 2026-03-09 19:20:36.838018 | controller | device_owner: compute:nova 2026-03-09 19:20:36.838135 | controller | device_profile: null 2026-03-09 19:20:36.838471 | controller | dns_assignment: 2026-03-09 19:20:36.838565 | controller | - fqdn: host-192-168-122-11.openstacklocal. 2026-03-09 19:20:36.838572 | controller | hostname: host-192-168-122-11 2026-03-09 19:20:36.838577 | controller | ip_address: 192.168.122.11 2026-03-09 19:20:36.838583 | controller | dns_domain: '' 2026-03-09 19:20:36.838682 | controller | dns_name: '' 2026-03-09 19:20:36.838774 | controller | extra_dhcp_opts: [] 2026-03-09 19:20:36.839000 | controller | fixed_ips: 2026-03-09 19:20:36.839097 | controller | - ip_address: 192.168.122.11 2026-03-09 19:20:36.839107 | controller | subnet_id: edcfb140-8934-4cc7-a46a-d8cf20ecfdf7 2026-03-09 19:20:36.839116 | controller | hardware_offload_type: null 2026-03-09 19:20:36.839198 | controller | hints: '' 2026-03-09 19:20:36.839324 | controller | id: 6d648f67-f72a-4d32-8184-c2d17ffb3d8c 2026-03-09 19:20:36.839449 | controller | ip_allocation: immediate 2026-03-09 19:20:36.839587 | controller | mac_address: fa:16:3e:08:be:f9 2026-03-09 19:20:36.839705 | controller | name: controller-593e5d4e-760b-40a9-aa62-62e7bf2adfde 2026-03-09 19:20:36.839818 | controller | network_id: d09b02e2-af90-41f7-a8a9-5064393c43d0 2026-03-09 19:20:36.839919 | controller | numa_affinity_policy: null 2026-03-09 19:20:36.840049 | controller | port_security_enabled: false 2026-03-09 19:20:36.840184 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-09 19:20:36.840276 | controller | propagate_uplink_status: null 2026-03-09 19:20:36.840368 | controller | resource_request: null 2026-03-09 19:20:36.840490 | controller | revision_number: 4 2026-03-09 19:20:36.840587 | controller | qos_network_policy_id: null 2026-03-09 19:20:36.840676 | controller | qos_policy_id: null 2026-03-09 19:20:36.840770 | controller | security_group_ids: [] 2026-03-09 19:20:36.840855 | controller | status: ACTIVE 2026-03-09 19:20:36.840936 | controller | tags: [] 2026-03-09 19:20:36.841044 | controller | trunk_details: null 2026-03-09 19:20:36.841197 | controller | trusted: null 2026-03-09 19:20:36.841321 | controller | updated_at: '2026-03-09T18:22:32Z' 2026-03-09 19:20:37.198726 | controller | changed 2026-03-09 19:20:37.265368 | 2026-03-09 19:20:37.265512 | TASK [Set port info yaml for further usage] 2026-03-09 19:20:37.388932 | controller | ok 2026-03-09 19:20:37.416788 | 2026-03-09 19:20:37.416875 | TASK [Detach port] 2026-03-09 19:20:49.708800 | controller | changed 2026-03-09 19:20:49.762326 | 2026-03-09 19:20:49.762461 | LOOP [Delete port association with trunk network] 2026-03-09 19:20:49.845463 | 2026-03-09 19:20:49.845666 | LOOP [Delete trunk sub ports (tagged ones)] 2026-03-09 19:20:49.984209 | 2026-03-09 19:20:49.984360 | TASK [Delete the trunk network itself] 2026-03-09 19:20:50.603359 | controller | skipping: Conditional result was False 2026-03-09 19:20:50.657645 | 2026-03-09 19:20:50.657769 | TASK [Remove the parent/trunk port itself] 2026-03-09 19:21:03.853060 | controller | changed 2026-03-09 19:21:03.884770 | 2026-03-09 19:21:03.884856 | TASK [Dettach router from subnet] 2026-03-09 19:21:04.535808 | controller | skipping: Conditional result was False 2026-03-09 19:21:04.569708 | 2026-03-09 19:21:04.569826 | TASK [Delete router] 2026-03-09 19:21:05.193112 | controller | skipping: Conditional result was False 2026-03-09 19:21:05.275523 | 2026-03-09 19:21:05.275657 | TASK [Delete subnet] 2026-03-09 19:21:17.488912 | controller | changed 2026-03-09 19:21:17.523957 | 2026-03-09 19:21:17.524103 | TASK [Delete network] 2026-03-09 19:21:34.697403 | controller | changed 2026-03-09 19:21:34.750277 | 2026-03-09 19:21:34.750415 | TASK [Remove clouds.yml secret] 2026-03-09 19:21:35.126627 | controller | changed 2026-03-09 19:21:35.216973 | 2026-03-09 19:21:35.217089 | PLAY RECAP 2026-03-09 19:21:35.217130 | controller | ok: 16 changed: 10 unreachable: 0 failed: 0 skipped: 8 rescued: 0 ignored: 0 2026-03-09 19:21:35.217151 | 2026-03-09 19:21:35.321924 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup.yml@master] 2026-03-09 19:21:35.331639 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml@master] 2026-03-09 19:21:35.913221 | 2026-03-09 19:21:35.913337 | PLAY [Report DLRN results after job run] 2026-03-09 19:21:35.991237 | 2026-03-09 19:21:35.991357 | TASK [Report job status to DLRN] 2026-03-09 19:21:36.054607 | controller | ok 2026-03-09 19:21:36.113578 | 2026-03-09 19:21:36.113716 | TASK [dlrn_report : Warning if DLRN is not running] 2026-03-09 19:21:36.188666 | controller | skipping: Conditional result was False 2026-03-09 19:21:36.213721 | 2026-03-09 19:21:36.213820 | TASK [dlrn_report : Install dlrnapi-client shyaml package] 2026-03-09 19:21:36.268682 | controller | skipping: Conditional result was False 2026-03-09 19:21:36.294827 | 2026-03-09 19:21:36.294950 | TASK [dlrn_report : Install kinit related package] 2026-03-09 19:21:36.349466 | controller | skipping: Conditional result was False 2026-03-09 19:21:36.374476 | 2026-03-09 19:21:36.374584 | TASK [dlrn_report : Install dlrn kerberos related packages] 2026-03-09 19:21:36.418819 | controller | skipping: Conditional result was False 2026-03-09 19:21:36.479431 | 2026-03-09 19:21:36.479553 | TASK [Set zuul-log-path fact] 2026-03-09 19:21:36.554258 | controller | skipping: Conditional result was False 2026-03-09 19:21:36.579433 | 2026-03-09 19:21:36.579528 | TASK [dlrn_report : Set the the value of cifmw_repo_setup_promotion when multiple dlrn tags are used] 2026-03-09 19:21:36.643986 | controller | skipping: Conditional result was False 2026-03-09 19:21:36.668440 | 2026-03-09 19:21:36.668541 | TASK [Get hash related data from repo_setup role] 2026-03-09 19:21:36.743103 | controller | skipping: Conditional result was False 2026-03-09 19:21:36.768306 | 2026-03-09 19:21:36.768396 | TASK [dlrn_report : Perform kinit for DLRN kerberos authentication] 2026-03-09 19:21:37.423301 | controller | skipping: Conditional result was False 2026-03-09 19:21:37.476602 | 2026-03-09 19:21:37.476718 | TASK [dlrn_report : Set empty value for dlrnapi password] 2026-03-09 19:21:37.552583 | controller | skipping: Conditional result was False 2026-03-09 19:21:37.587826 | 2026-03-09 19:21:37.587966 | TASK [dlrn_report : Report results to dlrn for the tested hash] 2026-03-09 19:21:38.242555 | controller | skipping: Conditional result was False 2026-03-09 19:21:38.434072 | 2026-03-09 19:21:38.434182 | PLAY RECAP 2026-03-09 19:21:38.434228 | controller | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 10 rescued: 0 ignored: 0 2026-03-09 19:21:38.434255 | 2026-03-09 19:21:38.571460 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml@master] 2026-03-09 19:21:38.578310 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master] 2026-03-09 19:21:39.178719 | 2026-03-09 19:21:39.178820 | PLAY [all] 2026-03-09 19:21:39.288317 | 2026-03-09 19:21:39.288449 | TASK [include_role : fetch-output] 2026-03-09 19:21:39.349569 | controller | ok 2026-03-09 19:21:39.356846 | crc | ok 2026-03-09 19:21:39.415540 | 2026-03-09 19:21:39.415629 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-09 19:21:39.546314 | controller | ok 2026-03-09 19:21:39.556819 | crc | ok 2026-03-09 19:21:39.580454 | 2026-03-09 19:21:39.580526 | TASK [fetch-output : Set log path for single node] 2026-03-09 19:21:39.619280 | controller | skipping: Conditional result was False 2026-03-09 19:21:39.640149 | crc | skipping: Conditional result was False 2026-03-09 19:21:39.694292 | 2026-03-09 19:21:39.694425 | LOOP [fetch-output : Ensure local output dirs] 2026-03-09 19:21:40.090656 | controller -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/logs/controller" 2026-03-09 19:21:40.097582 | crc -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/logs/crc" 2026-03-09 19:21:40.289708 | controller -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/artifacts" 2026-03-09 19:21:40.332869 | crc -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/artifacts" 2026-03-09 19:21:40.491624 | controller -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/docs" 2026-03-09 19:21:40.545706 | crc -> localhost | ok: "/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/docs" 2026-03-09 19:21:40.545878 | crc -> localhost | ok: All items complete 2026-03-09 19:21:40.545946 | 2026-03-09 19:21:40.582860 | 2026-03-09 19:21:40.583034 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-09 19:21:41.345339 | crc | changed: 2026-03-09 19:21:41.347807 | crc | .d..t...... ./ 2026-03-09 19:21:41.347920 | crc | >f+++++++++ crc-cloud-workdir-crc-all-logs.tar.gz 2026-03-09 19:21:41.347953 | crc | >f+++++++++ zuul-output-crc-all-logs.tar.gz 2026-03-09 19:21:41.347973 | crc | cd+++++++++ crc-cloud/ 2026-03-09 19:21:41.716179 | controller | changed: 2026-03-09 19:21:41.716281 | controller | .d..t...... ./ 2026-03-09 19:21:41.716308 | controller | >f+++++++++ zuul-output-controller-all-logs.tar.gz 2026-03-09 19:21:41.842791 | crc | ok 2026-03-09 19:21:42.189157 | controller | ok 2026-03-09 19:21:42.378819 | crc | ok 2026-03-09 19:21:42.667073 | controller | ok 2026-03-09 19:21:42.667212 | controller | changed: All items complete 2026-03-09 19:21:42.667248 | 2026-03-09 19:21:42.773483 | 2026-03-09 19:21:42.773593 | TASK [include_role : fetch-output-openshift] 2026-03-09 19:21:42.790813 | controller | skipping: Conditional result was False 2026-03-09 19:21:42.811978 | crc | skipping: Conditional result was False 2026-03-09 19:21:42.845274 | 2026-03-09 19:21:42.845347 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir] 2026-03-09 19:21:43.265263 | controller -> localhost | ok: Item: artifacts Runtime: 0:00:00.013782 2026-03-09 19:21:43.545048 | controller -> localhost | ok: Item: docs Runtime: 0:00:00.011691 2026-03-09 19:21:43.763332 | 2026-03-09 19:21:43.763430 | PLAY [all] 2026-03-09 19:21:43.847687 | 2026-03-09 19:21:43.847820 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes] 2026-03-09 19:21:44.530392 | controller | changed 2026-03-09 19:21:44.780432 | crc | changed 2026-03-09 19:21:44.993225 | 2026-03-09 19:21:44.993334 | PLAY RECAP 2026-03-09 19:21:44.993376 | controller | ok: 5 changed: 3 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0 2026-03-09 19:21:44.993404 | crc | ok: 4 changed: 2 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0 2026-03-09 19:21:44.993422 | 2026-03-09 19:21:45.096817 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master] 2026-03-09 19:21:45.106940 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master] 2026-03-09 19:21:45.742068 | 2026-03-09 19:21:45.742175 | PLAY [localhost] 2026-03-09 19:21:45.759325 | 2026-03-09 19:21:45.759404 | TASK [generate-zuul-manifest : Generate Zuul manifest] 2026-03-09 19:21:46.463043 | localhost | changed 2026-03-09 19:21:46.473242 | 2026-03-09 19:21:46.473402 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul] 2026-03-09 19:21:46.509690 | localhost | ok 2026-03-09 19:21:46.529196 | 2026-03-09 19:21:46.529335 | TASK [add-fileserver : Create SSH private key tempfile] 2026-03-09 19:21:46.953805 | localhost | changed 2026-03-09 19:21:46.958925 | 2026-03-09 19:21:46.958994 | TASK [add-fileserver : Create SSH private key from secret] 2026-03-09 19:21:47.705077 | localhost | changed 2026-03-09 19:21:47.716865 | 2026-03-09 19:21:47.717007 | TASK [add-fileserver : Add fileserver ssh key] 2026-03-09 19:21:48.141549 | localhost | Identity added: /var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/tmp/ansible.2mw4fmor (/var/lib/zuul/builds/e72ba1c8eae94a9da88ed53f3a62dcb0/work/tmp/ansible.2mw4fmor) 2026-03-09 19:21:48.141880 | localhost | ok: Runtime: 0:00:00.012462 2026-03-09 19:21:48.151773 | 2026-03-09 19:21:48.151929 | TASK [add-fileserver : Remove SSH private key from disk] 2026-03-09 19:21:48.529048 | localhost | ok: Runtime: 0:00:00.008447 2026-03-09 19:21:48.533966 | 2026-03-09 19:21:48.534054 | TASK [add-fileserver : Add fileserver to inventory] 2026-03-09 19:21:48.614037 | localhost | changed 2026-03-09 19:21:48.618666 | 2026-03-09 19:21:48.618743 | TASK [add-fileserver : Add fileserver server to known hosts] 2026-03-09 19:21:49.049139 | localhost | changed 2026-03-09 19:21:49.092917 | 2026-03-09 19:21:49.093134 | PLAY [localhost] 2026-03-09 19:21:49.113063 | 2026-03-09 19:21:49.113249 | TASK [Generate bulk log download script] 2026-03-09 19:21:49.136886 | localhost | ok 2026-03-09 19:21:49.157259 | 2026-03-09 19:21:49.157437 | TASK [local-log-download : Check API endpoint is defined] 2026-03-09 19:21:49.199420 | localhost | ok: All assertions passed 2026-03-09 19:21:49.206105 | 2026-03-09 19:21:49.206188 | TASK [local-log-download : Create download script] 2026-03-09 19:21:49.699456 | localhost -> localhost | changed 2026-03-09 19:21:49.710202 | 2026-03-09 19:21:49.710274 | TASK [Register quick-download link] 2026-03-09 19:21:49.739332 | localhost | ok 2026-03-09 19:21:49.755381 | 2026-03-09 19:21:49.755440 | PLAY [logserver.rdoproject.org] 2026-03-09 19:21:49.765327 | 2026-03-09 19:21:49.765386 | TASK [Set zuul-log-path fact] 2026-03-09 19:21:49.782733 | logserver.rdoproject.org | ok 2026-03-09 19:21:49.791895 | 2026-03-09 19:21:49.791957 | TASK [set-zuul-log-path-fact : Set log path for a build] 2026-03-09 19:21:49.830157 | logserver.rdoproject.org | ok 2026-03-09 19:21:49.836305 | 2026-03-09 19:21:49.836383 | TASK [upload-logs : Create log directories] 2026-03-09 19:21:50.692011 | logserver.rdoproject.org | changed 2026-03-09 19:21:50.699519 | 2026-03-09 19:21:50.699743 | TASK [upload-logs : Ensure logs are readable before uploading] 2026-03-09 19:21:51.086599 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.057066 2026-03-09 19:21:51.091345 | 2026-03-09 19:21:51.091407 | TASK [upload-logs : Upload logs to log server] 2026-03-09 19:21:56.564335 | logserver.rdoproject.org | Output suppressed because no_log was given 2026-03-09 19:21:56.570760 | 2026-03-09 19:21:56.570891 | LOOP [upload-logs : Compress console log and json output] 2026-03-09 19:21:56.638168 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-09 19:21:56.647497 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-09 19:21:56.657924 | 2026-03-09 19:21:56.658111 | LOOP [upload-logs : Upload compressed console log and json output] 2026-03-09 19:21:56.698934 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-09 19:21:56.699332 | 2026-03-09 19:21:56.702380 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-09 19:21:56.714268 | 2026-03-09 19:21:56.714462 | LOOP [upload-logs : Upload console log and json output]